We are grateful to the reviewer for the careful review and constructive comments.

Summary

First, we would like to kindly point out that the reviewer’s summary of our work is somewhat unfair and inaccurate. The reviewer claims that "the orthogonal component of the score diverges as $1/\sigma$ for $t \to 0$ while its tangential component approaches a constant" is a "well-known fact." This statement is unfair. The truly well-known fact is the divergence of the score and manifold overfitting phenomenon when the data is supported on manifolds, rather than this detailed statement about the behavior of its orthogonal and tangential components we provided. Although some earlier works such as [1], [3] implicitly hinted at the scale differences in spirit, the observation that "the orthogonal component of the score diverges as $1/\sigma$ for $t \to 0$ while its tangential component is of order 1" is explicitly and mathematically formulated by us for the first time in the known manifold setup, accompanied by rigorous mathematical proofs. Reviewers 42Gv and vZYH have also acknowledged that our result on the scale discrepancy is the first analysis in this field.

Regarding the experimental evaluation, the reviewer stated that "the increase in performance for toy models is small." We respectfully disagree with this assessment. We believe these experiments effectively demonstrate the validity of our proposed method and confirm the correctness of our theoretical analysis.

Strengths And Weaknesses

We have never claimed that "standard generative diffusion cannot deal well with manifold data", which seems to be the viewpoint that the reviewer is attempting to refute. Our actual argument is that mitigating the scale discrepancy of the score function can improve the performance of diffusion models, while it is true that diffusion models outperform likelihood-based methods.

The statement that "the 'irrelevance' of the tangential component of the score near zero is a feature, not a bug, as in that phase (manifold consolidation) the role of the score is to project the samples onto the manifold since the intrinsic distribution of the manifold has been determined at earlier times (manifold coverage phase)" can, at best, explain why diffusion models can work in manifold settings or outperform likelihood-based methods in such scenarios. However, the precision of the intrinsic/internal distribution on the manifold determined at earlier times (manifold coverage phase) may not be sufficient. The tangential component of the score near zero time (manifold consolidation phase) allows for a more refined characterization of the intrinsic distribution of the manifold and consequently is not an "irrelevant" part. This is one of the motivations behind our proposed methods, i.e., Niso and Tango. We believe that this statement may not be sufficient to serve as evidence against the arguments in our paper.

We believe that the reviewer has fully understood our motivation, as evidenced by their statement: "… have value in specific scenarios where the correct estimation of the tangential components at times close to zero is vital." However, we would like to argue that this is not limited to specific scenarios but rather represents a general setting.

In fact, if one truly aims to accurately compute the tangential component at times close to zero, we believe our methods effectively address this problem and lay the foundation for future studies. As shown in Figure 1 in our paper, our method significantly reduces the error of the tangential component at times close to zero, thereby improving overall accuracy.

We apologize for overlooking relevant literature and thank the reviewer for highlighting these important references. We greatly appreciate their insightful feedback and thorough knowledge of the field. Specifically, we find reference [3] interesting because its Jacobian eigendecomposition of the denoiser mapping also implicitly reflects scale differences, which aligns with the statements in our paper.

We will incorporate the following discussion of these papers in the related work section in the revised version.

The work [1] by Stanczuk et al. essentially uses the score discrepancy to identify the intrinsic dimension of the manifold. Kadkhodaie et al. [3] describe denoising in terms of basis functions, which implies the scale discrepancy we discussed. In particular, the Jacobian eigendecomposition of the denoiser mapping (Eq. (12)) implicitly captures these scale differences, aligning with Eq. (10) in our paper. However, our work explicitly reveals this phenomenon in manifold settings using more formal mathematical formulations and provides solutions to address it. References [4] [5] [6] study the fundamental principles of learning manifold-structured data distributions through Gaussian Mixture Models, Hidden Manifold Models, and Generalized Linear Models. References [1] and [2] also investigate the intrinsic geometry of generative diffusion models by studying the eigendecomposition of the Jacobian of the score functions. Moreover, we believe that our proposed methods can further improve the accuracy of diffusion models during the manifold consolidation phase as described in [2].

Questions: Real Data Case

Our current algorithm requires explicit knowledge of the manifold structure, which indeed limits its applicability to image or language data, where the manifold is often assumed but not explicitly known.

One possible approach to extending our algorithm is to train an Autoencoder to learn the manifold structure. Specifically, this involves learning an Encoder $\phi_{\theta_1}: \mathcal{M} \to \mathbb{R}^d$ and a Decoder $\psi_{\theta_2}: \mathbb{R}^d \to \mathcal{M}$, which serve as a parameterized representation of the manifold. The subspace spanned by $\nabla\psi_{\theta_2}(\phi_{\theta_1}(x))$ corresponds to the tangent space at point $x$ on the manifold. After obtaining a basis of the tangent space, we can further decompose the score function into its tangential and normal components. In principle, our method can also work on manifolds learned via an Autoencoder. However, in this work, we do not plan to implement this approach and instead leave it for future research.

We will add a discussion on this limitation in the main text (see next section).

Limitations

We will address this limitation explicitly in the main text. As follows:

Unlike previous works on distributions on a known manifold, our method avoids relying on geometric information like geodesics or heat kernels, greatly reducing computational complexity. However, compared to approaches under the manifold hypothesis, our method still requires knowledge of the manifold’s definition, including projection operators. In fields like image and language processing, the manifold structure is often assumed but not explicitly known. Future work will focus on extending our approach to scenarios where the manifold is undefined or implicit.

Final Words

Overall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function. If one aims to improve accuracy by precisely computing the tangential component near $t=0$, we believe our approach captures the essence of the problem and has the potential to inspire future research.

While we acknowledge that some relevant papers were overlooked in the current version, we believe the novelty of our approach is still evident. We will incorporate discussions of the related work in the revised version. Once again, we sincerely thank the reviewer for their constructive feedback.

We sincerely thank the reviewer for the thoughtful comments and valuable recommendations.

Strengths

We thank the reviewer for their positive evaluation of our work. By addressing the score discrepancies within and normal to the manifold, our analyses provide a novel contribution to the field. Leveraging manifold knowledge, we proposed two intuitive methods to tackle this problem effectively.

Weaknesses

W1: Manifold Assumption

Our current algorithm requires explicit knowledge of the manifold structure, which indeed limits its applicability to image or language data, where the manifold is often assumed but not explicitly known.

One possible approach to extending our algorithm is to train an Autoencoder to learn the manifold structure at first. Specifically, this involves learning an Encoder $\phi_{\theta_1}: \mathcal{M} \to \mathbb{R}^d$ and a Decoder $\psi_{\theta_2}: \mathbb{R}^d \to \mathcal{M}$, which serve as a parameterized representation of the manifold. The subspace spanned by $\nabla\psi_{\theta_2}(\phi_{\theta_1}(x))$ corresponds to the tangent space at point $x$ on the manifold. After obtaining a basis of the tangent space, we can further decompose the score function into its tangential and normal components. In principle, our method can also work on manifolds learned via an Autoencoder. However, in this work, we do not plan to implement this approach and instead leave it for future research.

W2: Baseline

Yes, we aim to use manifold information in these particular ways. The reviewer's suggestion is excellent and precisely reflects our existing approach. The results of the baselines (Iso or Iso+res) shown in the paper are conducted exactly as suggested by the reviewer. Specifically, we use the Reverse SDE sampling algorithm, which includes a projection step as the final step (see lines 205-207 in the main text). In other words, the Iso and Iso+res methods in our paper have already been combined with manifold information. We will make this information more explicit in the revised version of the paper.

Besides, we have conducted Riemannian diffusion models (RDM, [*]) on our experiments for comparison. This algorithm, based on sliced score matching and Hutchinson estimation, is applicable to general known manifolds. Therefore, we use RDM as a baseline and conduct experiments on Mesh data and SO(10).

Table 1 presents the comparison between the vanilla algorithm (with a final projection step), our method (using Niso+res with Reverse SDE), and RDM (baseline). Our method consistently outperforms the baseline. Specifically, in the SO(10) case, our approach outperforms RDM, while the vanilla algorithm does not. For Mesh data, RDM does not perform well, which can be explained by the fact that RDM relies on defining a diffusion process on the manifold to achieve a uniform distribution. Due to the geometric structure of the manifold, such as the narrow neck of "Spot the Cow", achieving a uniform distribution via SDEs on manifolds proves challenging.

Table 1: Computational errors for Mesh data and SO(10). The results are reported as the mean and standard deviation over five independent runs.

[*] Huang, Chin-Wei, et al. "Riemannian diffusion models." Advances in Neural Information Processing Systems 35 (2022): 2750-2761.

W3: Ablation Study

Following the reviewer’s suggestion, we have added ablation studies to evaluate the sensitivity of our method to parameters. In our algorithm, $\sigma_{\rm min}$, $c_{\rm niso}$, and $c_{\rm tango}$ are relatively important hyperparameters. Under the $SO(10)$ experiment, we present the computational errors for different parameter choices. Some of the results below are already contained in Fig. 3 in Appendix E of the paper.

Table 2: The impact of $\sigma_{\rm min}$ on the error under the Niso algorithm.

Table 3: The impact of $c_{\rm niso}$ on the error under the Niso algorithm.

Table 4: The impact of $\sigma_{\rm min}$ on the error under the Tango algorithm.

Table 5: The impact of $c_{\rm niso}$ on the error under the Tango algorithm.

These results indicate that reducing the hyperparameters does not significantly impact the performance of our method. Moreover, with different $\sigma_{\rm min}$ (see Table 1), our algorithm consistently outperforms the vanilla method (iso+res). We will include all these numerical results in our paper.

W4: Citation

We appreciate the reviewer’s suggestions and will incorporate discussions of these three references into Part 1 of the Related Work section. Specifically, we will include the following points:

Wenliang et al. show that the score perpendicular to noisy data of images is mostly Gaussian-like, whereas the score within the manifold has additional properties. The work by Stanczuk et al. essentially uses the score discrepancy to identify the intrinsic dimension of the manifold. Kadkhodaie et al. describe denoising in terms of basis functions, which implies the scale discrepancy we discussed. In particular, the Jacobian eigendecomposition of the denoiser mapping (Eq. (12)) implicitly captures these scale differences, aligning with Eq. (10) in our paper. However, our work explicitly reveals this phenomenon in manifold settings using more formal mathematical formulations and provides solutions to address it.

Questions

Q1: Hyperparameters

See the Weaknesses section (W3).

Q2: Combination of NISO and Tango

NISO and Tango handle the normal direction in fundamentally different ways: NISO seeks to relax the scale of the normal component, whereas Tango directly discards it. In the paper, NISO and Tango are introduced as two independent algorithms. It seems infeasible to effectively combine them as a new approach.

Limitations

We will address this limitation explicitly in the main text. As follows:

Unlike previous works on distributions on a known manifold, our method avoids relying on geometric information like geodesics or heat kernels, greatly reducing computational complexity. However, compared to approaches under the manifold hypothesis, our method still requires knowledge of the manifold’s definition, including projection operators. In fields like image and language processing, the manifold structure is often assumed but not explicitly known. Future work will focus on extending our approach to scenarios where the manifold is undefined or implicit.

Final Words

Overall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function.

While we acknowledge that some relevant papers were overlooked in the current version, we believe the novelty of our approach is still evident. We will incorporate discussions of the related work in the revised version. Once again, we sincerely thank the reviewer for their constructive feedback.

We appreciate the reviewer’s valuable questions and suggestions.

Strengths

We appreciate the reviewer for their positive evaluation of our work. Our study provides a detailed characterization of the score function behavior for both the original diffusion loss and the two proposed variants, each of which is well motivated and thoughtfully developed.

Weaknesses

W1: Smoothness of Manifolds

Indeed, when describing our algorithm, we assume the manifold to be smooth, and part of the mesh dataset satisfies this assumption. The manifold $\mathcal{M}$ corresponding to the mesh dataset is a polyhedron composed of triangles. On each face (triangle), the manifold is smooth, as it can be considered a part of a plane. The non-smooth points are the shared edges between adjacent faces, which form a zero-measure set. Therefore, this manifold is almost everywhere smooth, and our proposed algorithm is still applicable to this manifold without contradiction. In the implementation, for $x \in \mathcal{M}$, the normal direction of $x$ is the normal vector of the triangle (plane) that $x$ belongs to. The normal components are the projections of the score along this normal vector. For points lying on the shared edges (with zero probability of occurrence), we arbitrarily select the normal vector of one of the adjacent faces as the normal direction for that point. We will add more details about this example in the main text, as discussed above.

W2: Performance of Tango

We argue that Tango does not always underperform. In fact, our Tango algorithm still outperforms the vanilla diffusion model when using the annealed sampler. Moreover, in the dipeptide example, the optimal result was achieved with the Tango algorithm rather than Niso. At this stage, there is no definite conclusion regarding which method—Niso or Tango—is superior.

While Tango tends to be slower due to the annealing sampler, the runtime is primarily determined by the number of steps of the inner Langevin dynamics. For example, in the toy experiment R2inR3, the sampling time for Reverse SDE is 1.68 seconds, while for Annealing SDE, the sampling time increases to 5.62 seconds with an inner step of 5 and to 9.40 seconds with an inner step of 10. The first phase of Annealing SDE (see Algorithm 3) requires only 0.82 seconds. Overall, the computational cost of Annealing SDE is approximately 3–6 times higher than that of Reverse SDE. These measurements, conducted on a standard laptop, highlight the relative computational expense. A detailed discussion on computational costs will be included in the appendix.

W3: Baseline

Following the reviewer’s suggestion, we have conducted the algorithm in [1] on our experiments for comparison. Diffusion models constrained to the manifold often rely on complex geometric information, such as the heat kernel or logarithmic mapping, to compute transition probabilities, which limits the computation of the denoising score matching loss. In contrast, the algorithm from Riemannian diffusion models (RDM, [2]), based on sliced score matching (or implicit score matching in [1]) and Hutchinson estimation, is applicable to more general manifolds. Therefore, we use the RDM algorithm as a baseline and conduct experiments on Mesh data and SO(10). However, RDM failed in the Alanine dipeptide case, which we suspect may be due to the limitation of sliced score matching when applied to SE(3)-equivariant distributions.

Table 1 presents the comparison between the vanilla algorithm (with a final projection step), our method (using Niso+res with Reverse SDE), and RDM (baseline). Our method consistently outperforms the baseline. Specifically, in the SO(10) case, our approach outperforms RDM, while the vanilla algorithm does not. For Mesh data, RDM does not perform well, which can be explained by the fact that RDM relies on defining a diffusion process on the manifold to achieve a uniform distribution. Due to the geometric structure of the manifold, such as the narrow neck of "Spot the Cow", achieving a uniform distribution via SDEs on manifolds proves challenging.

Table 1: Computational errors for Mesh data and SO(10). The results are reported as the mean and standard deviation over five independent runs.

We appreciate the reviewer’s suggestion and hope that these additional experiments will be taken into account when reevaluating our work. We will include these additional experiments in the revised version of the paper.

Minor Weaknesses

W4: Bias-Variance Trade-off

We thank the reviewer for carefully reading our appendix. As one of the fundamental philosophyies in machine learning, the bias-variance tradeoff phenomenon is widely observed in parameter selection scenarios. In this ablation study, we demonstrate that our algorithm is still favorable over unmodified diffusion. We do not consider this a weakness but rather evidence further supporting the advantages of our algorithm.

W5: Non-isotropic Noise

In the implementation, we chose $\sigma_t^{\alpha_t} = c_{\rm iso}$ as a constant, which aligns with the reviewer's understanding. However, our approach is not entirely equivalent to pre-adding noise to the data before training, as our loss function (see Eq. (19)) employs non-isotropic score matching. The general formulation in the theorem was introduced to rigorously prove the scale differences from a mathematical perspective. While it is possible that selecting a more flexible $\alpha_t$ might lead to better results, for simplicity in our numerical experiments, we only investigated the case where $\alpha_t = \log c_{\rm iso} / \log \sigma_t$ in the paper.

Questions

Q1：'Almost surely'

In the appendix, we prove that $\ell_{\text{tango}}(t,\theta) = \mathbb{E}_{x, x_t} \|s^{\parallel}_{\theta}(x_t,t)-P(x_t)\nabla_{x_t}\log p_{\sigma_t}(x_t)\|^2 + C_1$.

Thus, when $s$, $p$, and $P$ are continuous functions, $s^{\parallel}_{\theta}(x_t,t) = P(x_t)\nabla_{x_t}\log p_{\sigma_t}(x_t)$ always holds, not just "almost surely." We will remove the phrase "almost surely" in the updated version.

Q2: Learned Manifold Case

In principle, our method can be extended to learned manifolds. As suggested by the reviewer, we use the AutoEncoder as an example. At first, we train an Encoder $\phi_{\theta_1}: \mathcal{M} \to \mathbb{R}^d$ and a Decoder $\psi_{\theta_2}: \mathbb{R}^d \to \mathcal{M}$, which provide a parameterized representation of the manifold. The subspace spanned by $\nabla\psi_{\theta_2}(\phi_{\theta_1}(x))$ corresponds to the tangent space $T\mathcal{M}_x$ at point $x$ on the manifold. Once the basis of the tangent space is obtained, the score function can be further decomposed into its tangential and normal components, which enables the implementation of our proposed methods. Intuitively, the success of our method on learned manifolds hinges on the Autoencoder's ability to accurately capture the underlying manifold structure. However, in this work, we do not plan to implement this approach and instead leave it as a direction for future research.

Q3: f(x,t) Term

In this paper, we focus on the Variance Exploding SDE (VESDE), characterized by $f(x, t) = 0$. To ensure clarity for readers, we will explicitly highlight this in the explanations of equations (21) and (22).

Final Words

Overall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function. If one aims to improve accuracy by precisely computing the tangential component near $t=0$, we believe our approach captures the essence of the problem and has the potential to inspire future research.

Once again, we sincerely thank the reviewer for their constructive feedback.

We appreciate the reviewer’s valuable feedback and suggestions.

Strengths

We thank the reviewer for their positive feedback on our work. In Theorem 3.1, we exploit the observation to analyze the behavior of the true score both close to and on the manifold, allowing for a neat decomposition of the diffusion loss functions. The proposed algorithms, derived directly from the theoretical insights, demonstrate strong empirical performance.

Weaknesses

W1: Baseline

Following the reviewer’s suggestion, we have conducted Riemannian diffusion models (RDM, [*]) on our experiments for comparison. This algorithm, based on sliced score matching and Hutchinson estimation, is applicable to general known manifolds. Therefore, we use RDM as a baseline and conduct experiments on Mesh data and SO(10). RDM failed in the Alanine dipeptide case, which we suspect may be due to the limitation of sliced score matching when applied to SE(3)-equivariant distributions.

Table 1 presents the comparison between the vanilla algorithm (with a final projection step), our method (using Niso+res with Reverse SDE), and RDM (baseline). Our method consistently outperforms the baseline. Specifically, in the SO(10) case, our approach outperforms RDM, while the vanilla algorithm does not. For Mesh data, RDM does not perform well, which can be explained by the fact that RDM relies on defining a diffusion process on the manifold to achieve a uniform distribution. Due to the geometric structure of the manifold, such as the narrow neck of "Spot the Cow", achieving a uniform distribution via SDEs on manifolds proves challenging.

Table 1: Computational errors for Mesh data, and SO(10). The results are reported as the mean and standard deviation over five independent runs.

We appreciate the reviewer’s suggestion and hope that these additional experiments will be taken into account when reevaluating our work. We will include these additional experiments in the revised version of the paper.

[*] Huang, Chin-Wei, et al. "Riemannian diffusion models." Advances in Neural Information Processing Systems 35 (2022): 2750-2761.

W2: Relation to Paper [1]

We thank the reviewer for pointing out this highly relevant paper and its similarities to our work. Indeed, the idea of adding noise along the normal direction is similar between the two studies. We will include the following discussion in a revised version.

Paper [1] proposes adding noise along the normal direction to "inflate" manifolds, which is similar to our niso method. Theorem 4 and Proposition 7 in [1] analyze the relationship between the perturbed distribution and the original distribution on the manifold, which aligns with conclusions implied by our equations (12) and (18).

On the other hand, there are several key differences. In [1], the added noise has a constant magnitude, requiring additional assumptions (e.g., Q-normally reachable) to prevent interference between noise at different points. In contrast, our analysis (Theorem 3.1) allows the additional noise magnitude to approach zero, relying on weaker assumptions. Additionally, [1] applies noise inflation prior to training the normalizing flow, whereas our method integrates noise addition directly into the diffusion process, aligning with the noise dynamics of diffusion models. Finally, while the analysis in [1] focuses on the perturbed density function, our work studies the asymptotic behavior of the score function.

W3: Additional References

We thank the reviewer for pointing out these related works. We will include citations to these papers in the Related Work section.

Additional comments

We thank the reviewer for their careful reading of our paper. We will make the necessary revisions accordingly. The operation $\wedge$ is defined as $a \wedge b = \min[a, b]$, and the definition of $\nabla^{\mathcal{M}}$ is as follows: $\nabla^{\mathcal{M}}f(x) := P(x)\nabla_{x}f(x)$ for any function $f$.

Questions

Q1: Refrasing

This section can indeed be written more clearly. In the revised version, we will rephrase the sentence as follows:

By carefully selecting $f$ and $g$, the distribution $p_{T}$ can be well approximated as a Gaussian distribution $\mathcal{N}(0, \sigma_T^2 I)$ at time $T$, where $\sigma_T^2$ is the variance of $X_T$.

Q2: Explanation of Figure 1

The tangential component corresponds to the first two coordinates (rather than the third) of the learned score function, as the manifold constraint ensures that the third component is zero. The tangential component is the first two coordinates of the learned score function. From a theoretical perspective, due to the scale discrepancy we mentioned, accurate estimation of the tangential component becomes harder as $t \to 0$, resulting in increased error. The experimental results are consistent with this theoretical analysis: the average error increased as $t \to 0$. Our method (the bottom two plots) reduces this error. The magnitude of the error is not related to the funnel structure. The funnel structure reflects the effective range of the distribution. As $t \to 0$, the distribution concentrates near $z = 0$, which leads to the funnel shape.

Q3: Utilized Metrics

In our setting, the data is strictly distributed on a submanifold of the space $\mathbb{R}^n$. Ideally, metrics for evaluation should compare distributions based on distances defined on the manifold itself. However, such computations are generally infeasible. As a practical alternative, we calculate distances between distributions in the entire space (with the exception of JS divergence for mesh data, which effectively captures differences on the manifold). Due to the unique properties of each manifold and the distributions defined on them, no single metric can reliably evaluate performance across all datasets. For instance, in the case of $\mathrm{SO}(10)$, which is embedded in a high-dimensional space, the 1-Wasserstein distance in $\mathbb{R}^n$ is computationally expensive and does not effectively capture distribution differences on the manifold, making sliced Wasserstein a better choice. Besides, for $\mathbb{R}^2$ embedded in $\mathbb{R}^3$, we observe that the variance of the 1-Wasserstein distance is large, while the variance of MMD is more stable.

Q4: Assumptions in Theorems

Our conclusions hold for points on the manifold, so in the proof, we only require local boundedness. Specifically, the following assumption is sufficient:

To make the theorem statement more concise, we adopted a stronger assumption:

For compact manifolds, the uniform boundedness assumptions always hold, and consequently, the local boundedness assumptions hold. We can always assume that the manifold is bounded, as the sample data we obtain always have finite values.

Limitations:

We will include a discussion on the assumptions of Theorem 3.1 and Theorem 4.1 in the paper, as mentioned in the reply to Question 4 above.

Final Words

Overall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function.

While we acknowledge that some relevant papers were overlooked in the current version, we believe the novelty of our approach is still evident. We will incorporate discussions of the related work in the revised version. Once again, we sincerely thank the reviewer for their constructive feedback.

Thank you for your response and careful consideration of my points. I appreciate the additional experiments and think they add value to the paper. I am also satisfied with the revised literature review. I am happy to adjust my score accordingly.