Thank you for the positive review. We appreciate that you found our theoretical contributions to be correct and that you agree we are tackling an important problem. Your main concerns appear to be centered on the experimental scope and the practical implications of our approach to the manifold hypothesis. We hope the following clarifications are helpful.

Manifold hypothesis

The classical manifold hypothesis assumes that there is a single, ground-truth manifold that one aims to recover (for example, a "true" manifold of images or proteins). This is how most of the theoretical analysis of diffusion models up to this point treat the problem, and also the way in which we analyse the setting up to Section 4. Afterwards, we offer an alternative perspective: we argue that in high dimensions there are many plausible interpolating manifolds given sparse observations. The choice of which manifold is learned is therefore determined by the inductive biases of the training procedure.

Practitioners already leverage this kind of intuition, though often in an ad-hoc manner, by experimenting with different model architectures that encode certain invariances. Our work takes a step toward understanding how to design such a bias purposefully if the desired manifold structure were known. We demonstrate this on toy examples (Figures 1 and 3), where different biases lead to different interpolating manifolds for the same set of data points. For a more complex and realistic case like MNIST, we use a VAE to construct a manifold, and guide the smoothing to occur along the manifold structure to examine its induced effects.

Weaknesses

• We think that designing the correct inductive biases for specific, complex datasets is a challenging task. We hope our work has helped make a step in this direction. Our primary goal with this work is to deepen the theoretical understanding of what an ideal bias would entail if one already knew the target manifold structure. We believe this is a necessary step before such principles can be used to design application-specific biases. For instance, in scientific machine learning, where data may be solutions to a given PDE, we are hopeful that this framework can be used to design more effective biases/models in follow-up work.
• To clarify, we are not claiming that diffusion models work only on manifold-structured data. Instead, we argue that they remain effective even in this challenging setting, and we posit that this effectiveness is linked to the implicit log-domain smoothing.
• You are correct that the projection operator is not implementable without prior knowledge of the target manifold. In our experiments, we use the knowledge of the true manifold (constructed explicitly in Section 5.1, and via a VAE in Section 5.2) to construct the smoothing mechanism, while the function that is being smoothed is the empirical score and thus unaware of the true manifold structure. These experiments are designed to illustrate that the type of smoothing can provide a mechanism that impacts the resulting interpolating manifold, as suggested by the results in Section 4. We do not propose this as a practical algorithm or argue that this is necessarily the right way to actually smooth your neural network. We discuss this in lines 316-320, noting that if manifold-adapted smoothing occurs in practice, it will have arose implicitly from the models inductive biases, via architectural choices such as convolutions and attention [1].

[1] Kamb and Ganguli, 2024, "An analytic theory of creativity in convolutional diffusion models"

Questions

• Biomedical datasets can vary greatly, but we do believe that many of them do satisfy some kind of manifold hypothesis, and that in these cases the manifold constraints might even be a bit more explicit than for images, making it possible to think about the right biases. We think an analysis in terms of privacy metrics is an important direction. However, it requiress a considerable amount of theoretical work which we plan to do in future publications. We are very interested in the reviewer's ideas for follow-up works on privacy in case the reviewer wants to elaborate.
• If one could learn a manifold structure via a global coordinate chart $\mathbb{R}^d \to \mathbb{R}^D$, with $\mathbb{R}^D$ the ambient space, then it would be possible to run a diffusion model in these coordinates (see [1]). However, standard formulations of Diffusion Maps (and to the best of our knowledge Laplacian Eigenmaps as well) do not yield a global feature map $\mathbb{R}^d \to \mathbb{R}^D$. While the inverse map $\mathbb{R}^D \to \mathbb{R}^d$ (from the embedding to latent space) can often be extended, extending the forward direction chart $\mathbb{R}^d \to \mathbb{R}^D$ is typically non-canonical and non-trivial. Still, the reviewer may be interested in [2], which uses diffusion maps in a generative setting with an algorithm similar to diffusion models.
• Yes indeed, in cases where the manifold is known, one can also run the whole algorithm directly on that manifold which has been explored in prior work, see for example [1]. However, our theoretical model is focused on analyzing why standard diffusion models work so well even when this structure is not explicitly known or enforced. Our work aims to make the connection between an inductive bias (e.g., tangential smoothing) and the recovered manifold more explicit.

[1a] De Bortoli; Mathieu; Hutchinson; Thornton; Teh; Doucet. 2022. “Riemannian Score‑Based Generative Modelling.”

[1b] Huang; Aghajohari; Bose; Panangaden; Courville. 2022. “Riemannian Diffusion Models.” Universität de Montréal & University of Toronto & McGill University

[2] Gottwald; Li; Marzouk; Reich. 2024. “Stable generative modeling using diffusion maps.”

Summary

We thank the reviewer for their questions, as clarifying these points has greatly helped improve the quality of the paper. We have modified the camera-ready version in several places accordingly. We would be more than happy to discuss any questions further or to hear if you feel that any of the points we mentioned would in your opinion help improve our camera-ready version.

In case you think that the above modifications have improved our rating beyond a borderline accept, we would be most appreciative if you would consider raising your score.

Thank you for your positive assessment and for providing such constructive feedback!

We appreciate the opportunity to clarify the presentation of our work, especially related to Assumptions 3.4 and 3.5. We have incorporated your feedback (and utilized the additional page) to improve the camera-ready version. We have expanded Section 4 with more intuition and enhanced the presentation in several places. As we cannot upload a revised manuscript at this stage, we will outline the changes below, focusing on the points you highlighted.

Assumption 3.4: Manifold Reach

We agree that the definition of reach would benefit from greater clarity. A minor error in the current definition may have contributed to some confusion. In the revised version, we have replaced this with a more intuitive, equivalent definition: the reach is the maximal distance from the manifold within which every point has a unique projection onto that manifold. This is the precise assumption used in our proofs and we believe this more intuitive definition may help clear some confusion.

To offer further intuition on the concept and its necessity: for a linear submanifold, any point in the ambient space has a unique closest point on the submanifold. This is not guaranteed for curved manifolds. Consider a circle, for which the center point is equidistant from every point on its circumference. The reach of this circular manifold is precisely its radius. Our assumption is designed to exclude ill-behaved manifolds with rapidly oscillating curvature (e.g., the graph of y=sin(1/x) near x=0), where the reach can become arbitrarily small locally, posing challenges for analysis. We are excluding this case with this assumption.

In the camera-ready version, we have moved the revised, more intuitive definition of reach into the main paper and included some of this intuition.

Assumption 3.5: Smoothing tangential to the manifold

This assumption bounds the extent to which the smoothing kernel operates in directions normal to the manifold. The linear case is again instructive. Imagine the manifold is a linear hyperplane. If a smoothing kernel $k_x$ only distributes mass parallel to this hyperplane, then for any point $Y \sim k_x$, we have $\text{dist}(Y,\mathcal{M})=\text{dist}(x,\mathcal{M})$. In this scenario, the assumption is trivially satisfied. However, if the kernel also smooths along directions normal to the manifold, the value of $\text{dist}(Y,\mathcal{M})$ will vary. The assumption specifies that this variation is bounded.

In the nonlinear case, the ambient space can be locally decomposed into a set of "parallel" manifold slices, at least within the reach (per Assumption 3.4). The same intuition applies: we require a global bound on how much mass the kernel assigns to these parallel slices, which is an important condition for our proof technique.

Theorem 4.1

We agree with the reviewer that the definition of the manifold family would have benefited from further discussion. A smooth compact manifold $\mathcal{M}$ is a member of this family $\mathbb{M}_\mu$ if the projection of the data distribution onto $\mathcal{M}$ has a density that is uniformly lower-bounded away from zero:

$$\inf_{x \in \mathcal{M}} p_{\mu, \mathcal{M}}(x) > 0.$$

The main intuition having no areas with $0$ probability is that these areas would never carry any samples, and could not be identified even if we have an infinite amount of samples. The requirement that the inf is strictly bounded away from $0$ is necessary to guarantee that even with a finite number of samples we can recover the whole manifold with a lower bounded probability. This type of assumption is typical in the literature on diffusion models and manifold learning (e.g., [1], [2]).

[1] Aamari. 2017. “Convergence Rates for Geometric Inference.” Université Paris-Saclay.

[2] Azangulov; Deligiannidis; Rousseau. 2024. “Convergence of Diffusion Models under the Manifold Hypothesis in High-Dimensions.”

Summary

We thank the reviewer again for their concrete feedback, which has helped us identify areas where our presentation could be improved. We have already integrated these changes into our camera-ready version. If any of the above explanations was particularly helpful, we would be very happy if you point that out so that we can elaborate on it in the camera ready version. If you feel like these explanations address your concerns and raise the papers appropriate rating beyond a borderline accept, we would be delighted if you could change your score accordingly.

Thank you for the positive feedback on our paper. We are very happy that you appreciate our theoretical contributions. We will address your points and questions below.

Weaknesses

We agree that Equation (5) is an assumption. However, we want to note that it is essentially the assumption that the learned score is some smoothing of the empirical score, rather than a restriction on the type of smoothing that can occur. Since we allow the kernels to be position-dependent (i.e., the kernels $k_x$ can be completely unrelated for different $x$), this assumption posits that neural network takes the target function and at each point $x$, smooth the function arbitrarily and completely independent of other points. This allows for a very large class of smoothing operators (or even non-smoothing operators if $k_x$ is a dirac) to be encompassed in this setting. We then study which kind of assumptions we have to make for $k_x$ to recover a specific target manifold (in particular, Assumption 3.5). It turns out that smoothing in the log-domain is already a very powerful tool to recover manifolds; moreover, by smoothing along tangential directions of a target manifold $\mathcal{M}$, one can induce a biases towards that particular interpolating manifold.

Regarding the practicality of our work, we see its primary contribution as deepening the theoretical understanding of how diffusion models interact with manifold-structured data. Nevertheless, we believe there are two key practical takeaways. First, smoothing empirical densities in the log-domain is more compatible with manifold structures than traditional density-domain smoothing (as illustrated in Figure 2), a principle that could inform the design of novel algorithms. Second, we argue one can induce a geometric bias by smoothing along the tangential directions of a desired manifold. While engineering neural networks with such specific inductive biases is a complex challenge, we hope our work provides a guiding intuition for this direction, which we plan to explore in future work.

Questions

• You raise an excellent point regarding the intuition of log-domain smoothing. In the theoretical limit where the log-density is $-\infty$ off-manifold, any smoothing kernel with support extending beyond the manifold would indeed yield a value of $-\infty$. However, for any positive time $t>0$, the noising process ensures the density $p_t(x)$ is positive everywhere. Consequently, the log-density is finite across the space. While smoothing with points far from the manifold will lower the log-density value on the manifold, the same effect occurs for all points. Since the final density will be normalized to mass 1, what matters isn't the absolute/specific values taken by the unnormalized smoothed log-density, but rather their relative differences. Our intuitive argument is that this procedure "punishes" points further away from the manifold more than points on the manifold, thus preserving the structure. We hope this clarifies the intuition. We have made sure to clarify this point in the paper and to emphasize that it is the relative effects that matter. We are open to making more changes if you think it would improve the presentation.
• You are correct about the definition of reach. We made a small error when finalizing the paper; the definition in the submitted version is incorrect. The proper definition is indeed equivalent to requiring a unique projection onto the manifold. This definition is more intuitive and we have changed it.
• When we say "with high probability", we mean that for any $\delta>0$, there exists an $N_\delta$ such that for $N>N_\delta$ the statement holds with probability at least $1-\delta$. We have revised the phrasing of the theorem to clarify this and to provide precise dependencies of the relevant quantities on each variable; please let us know if you would like more information regarding this point.

Summary

We want to thank the reviewer again for the great questions and we hope that our answers helped to address your concerns. We would be grateful for any final suggestions on which of these clarifications you feel are most helpful to incorporate into the final manuscript. If you think that our changes justify a change in your score, we would of course be delighted.

We would like to express our sincere gratitude to the reviewer for this rating and for the insightful, encouraging comments. We are very happy that you found our paper interesting! The references you provided are indeed highly relevant, and we have integrated them into the revised manuscript to better contextualize our work for the reader.

Other transforms than the logarithm

This is an excellent observation, and we agree that the core intuition extends beyond the logarithmic transform. We find this prospect very interesting. Our focus on the logarithm was motivated directly by the formulation of contemporary diffusion models: a neural network is trained to approximate $\nabla \log \hat{p}_t$. Since the network's approximation (and thus its implicit smoothing effect) occurs at the logarithmic level, we felt this was the natural choice for our analysis.

Our current paper was primarily focused on trying to help explain part of the success of modern diffusion models, and as such we haven't found a good experiment for more general functions $f$ in explaining this. However, we would greatly appreciate any suggestions or ideas that you may have in this direction.

We do, however, believe your point is especially prescient when considering the design of new algorithms. The principle that any monotonic function where $f(0) = -\infty$ could theoretically confer a similar geometry-adaptive benefit is a powerful one. Training a diffusion model with a different transform would mean that one would need to learn $\nabla f(p(x))$ and then transform this to $\nabla \log p(x)$, which is then used as a drift (which is possible if $f$ is an invertible function). This is an interesting direction to think about.

Synthetic and MNIST experiments

Thank you for the great suggestion. We are working on a 2D experiment to add to our camera-ready version.

Minor

Yes, we meant the convex hull and we have clarified this in the paper. Thank you very much for catching the other typos as well; they have been corrected.

Summary

We want to warmly thank you for your rating and your great questions! Adding a 2D experiment will be a significant improvement, and your observation about the broader class of applicable transformations is a very pleasing and interesting insight that we will continue to reflect on.