Generalization in diffusion models arises from geometry-adaptive harmonic representations

The paper studies the inductive bias of the models which involve a denoising step, i.e., score-based diffusion algorithms. To do so, the authors take a spectral approach and analyze the eigenspace and eigenvalues of the corresponding DNN denoiser Jacobian. The authors conjecture that the DNN denoiser are implicitly biased toward so called geometry-adaptive harmonic bases (GAHBs). The authors first show that such biases emerge on a set of synthetic $C^{\alpha}$ images. Further strengthening a point, the authors empirically show that the bias still emergences similarly even in suboptimal scenario (disk images and shuffled CelebA).

Diffusion generative models that use denoising DNNs have surpassed previous methods of learning probability models from images. The authors introduce a methodology to evaluate the properties of the trained denoiser and the density from which the data are drawn. They show empirically that diffusion models can converge to a unique continuous density model that is independent of the specific training samples. The convergence exhibits a phase transition between memorization and generalization as training data grows. They demonstrate that two denoising DNNs trained on non-overlapping subsets of a dataset learn nearly the same score function and, thus, the same density, indicating the existence of powerful inductive biases in the DNN architecture and/or training algorithm. The inductive bias of the network appears through the best basis, which is a geometry-adaptive harmonic basis (GAHB) when trained on photographic images. The DNN denoisers achieve near-optimal performance for the Cα class of images. They also investigate the inductive biases that enable this rapid convergence and show that DNN denoisers perform poorly for distributions whose optimal bases are not GAHB. For images drawn from low-dimensional manifolds, DNN denoisers achieve an accurate basis for the subspace and incorporate GAHB vectors in the remaining unconstrained dimensions. The denoiser performs a shrinkage operation on a basis adapted to the underlying image, which reveals oscillating harmonic structures along contours and inhomogeneous image regions. The paper leaves important open questions regarding the formal mathematical definition of this larger class of GAHB bases and how they result from the DNN computational architecture.

The authors address recent concerns regarding the memorization of denoising DNNs for generative image modeling by showing that two networks trained on disjoint datasets converge to the same score/density and hence generate similar images. Those experiments are supported by theoretical derivations clearly relating the inductive bias of the denoiser to that of the density. Leveraging recent work, e.g., (Monoho et al.,2020), the authors elucidate the sparse optimal basis, adapted to the geometry of the input image, along which denoising can be understood as a shrinkage operation. Furthermore, the authors validate and connect the results by evaluating the inductive bias on image classes with known optimal bases.

The authors study the issue of generalization and memorization in diffusion models. Diffusion models have as a backbone trained denoisers, and it has been observed that the use of powerful deep networks as denoisers can lead to situations where diffusion models memorize their training data (rather than being able to generate novel images from the high-dimensional data distribution), whereas in other cases the models seem to generalize. The authors study this for a specific class of deep network denoisers (bias-free CNNs) both empirically and theoretically. Empirically, they show that training these denoisers on CelebA and LSUN bedrooms for varying training set sizes witnesses a clear transition (qualitative and quantitative) between "memorization" performance of the trained diffusion model when the training set size is small, and "generalization" performance when it is sufficiently large. The authors hypothesize that the ability to generalize with relatively few samples from the high-dimensional distribution is due to inductive biases in the deep network denoiser, and in particular they make a connection with classical ideas on denoising from harmonic analysis to posit that DNN denoisers are biased towards "geometry-adaptive harmonic bases" (eg bandlets/wedgelets). They test this hypothesis empirically, demonstrating that these trained denoisers learn bandlet-like bases in which to perform shrinkage on toy image classes where such bases are optimal (horizon classes), and moreover that they persist in learning these types of bases even for classes of signals where they are suboptimal (image articulation manifolds).

This paper investigates the properties of memory and generalization in diffusion models, both experimentally and theoretically. Specifically, the paper identifies the properties of data sets that contribute to the transition phenomena of memory and generalization, and then explains this using a concept that extends the harmonic basis. The results and presentation of the paper are clear and the intent is well communicated to the reader. The reviewers agreed that the paper should be highly commended.

Accept (oral)