The Geometry of Phase Transitions in Diffusion Models: Tubular Neighbourhoods and Singularities

Thank you for your thoughtful and detailed review. We appreciate your deep understanding and positive feedback on our theoretical framework and experimental results, as well as your constructive suggestions. Below, we address your main points and questions.

Weakness:
As you pointed out, the appearance of a singularity in our framework corresponds to the collapse of a part of the data manifold to a single point at time $t$, which is analogous to the bifurcation of an isolated fixed point in a Curie-Weiss type transition. At the same time, our framework deals with phase transitions occurring in non-equilibrium phenomena in the form of diffusion processes, and we believe that it is necessary to extend the existing phase transition theories that have been developed to explain phase transitions in equilibrium systems.

Guided by reviewer TUbi's comments, we recognized the need to develop a more complete and rigorous theory of phase transitions in physics and the occurrence of phase transitions in diffusion processes. We look forward to continued discussions, including possible collaboration.

Connection to Fixed-Point Analysis (Weakness, Q1):
We agree that establishing a stronger theoretical connection between our geometric method and the fixed-point analysis in Raya and Ambrogioni (2023) would be highly insightful. Our approach indirectly incorporates the idea of fixed points by analyzing the behavior of $\nabla u$ at the boundary of the tubular neighborhood (Proposition 4.2). Specifically, the change in the proportion of particles within the tubular neighborhood, $\Gamma(t)$, corresponds to the integral of $p_t(x) \nabla u_t(x)$ over the boundary, providing an analog to the fixed-point framework in terms of analysis of free energies.

While Raya and Ambrogioni (2023) focus on the origin as a fixed point under ideal conditions, our framework naturally extends to scenarios with varying curvatures by leveraging the injectivity radius. This allows us to capture phase transitions at the boundary where the data manifold approximation breaks down. For instance, in a simple 1D example where the manifold consists of two points $\{1, -1\}$, the boundary of the tubular neighborhood is $\{0, 2, -2\}$. In the hypersphere case, the boundary is a union of the origin and a larger sphere of radius 2.

In our framework, $\Gamma(t)$ aligns with the fixed-point dynamics in some sense. We acknowledge the potential for deeper connections between our approach and the critical transitions in physics and plan to explore this in future work. If you are interested, we would welcome collaboration to further develop this connection.

Multiple Critical Times and Curvature (Weakness, Q2):
Your suggestion to study multiple critical times arising from varying local curvatures is well-taken. In our current work, we demonstrate this phenomenon using an experiment with disjoint arcs of different radii (Section 5.3). However, we acknowledge that this is only a preliminary step. To provide more robust evidence, we plan to conduct additional experiments with varied curvatures and datasets that combine multiple local geometries. This will allow us to better illustrate how separate transition times arise in such settings.

Experimental Variations and Statistical Physics Framework:
We agree that connecting our findings to the language of statistical physics would enhance the conceptual depth of our work. While this was beyond the scope of the current paper, we see this as a promising direction for future research and appreciate your suggestion.

Thank you once again for your very insightful feedback. We will revise the manuscript to address these points and include the proposed experimental extensions.

We sincerely appreciate your constructive comments and your support of our submission. We intend to investigate this direction in future research and believe it could further enhance the relevance and impact of our contributions.

Thank you for your detailed feedback and constructive suggestions. Below, we address your concerns point by point.

Experimental Results and Scope (Weakness 1):
We acknowledge that some experimental results, such as Fig. 5, may not seem to perfectly align with the theoretical predictions. However, this represents an important insight worth noting. The experiments in Fig. 5 demonstrate that this behavior, as discussed in Sec 5.2, arises from non-uniform curvature (Fig. 5(b), (c)). For cases like Fig. 5(a), the misalignment stems from the distance between the initial distributions in the forward and reverse processes. Additional results in Appendix J.3 support this analysis, showing alignment when the Gaussian noise variance is reduced, which is why the discrepancy occurs. We will also consider conducting additional experiments with datasets exhibiting varying curvatures to further validate the importance of the injectivity radius, as suggested.

Algorithm 1 Description (Weakness 2):
We apologize for the lack of clarity in Algorithm 1. A detailed explanation is provided in P26. To enhance comprehensibility, we will include intuitive explanations for each step in the appendix. Please let us know if there are specific aspects requiring further clarification.

Definition of Wasserstein Distance (Weakness 3):
The Wasserstein distance in our experiments is computed between the initial distribution of the forward process, $p_0(x)$, and the final distribution of the reverse process, $q_0(x)$, for each delayed initialization time $T$. We will revise the captions in relevant figures to clarify this.

Correlation Between Proportion and Wasserstein Distance (Q1):
It is known that in the final stages of the generation process, score vectors on an $\epsilon$-neighborhood are orthogonal to the tangent plane of the data manifold (refer to Stanczuk et al. (2024)). For vectors within a tubular neighborhood to be orthogonal to the tangent plane of the data manifold, they must be within the injectivity radius (see Theorem D.1 in Stanczuk et al. (2024) or Proposition 4.7 in Section 4). Therefore, particles are effectively drawn into the tubular neighborhood.

In fact, as shown in Fig. 4, the red line drops sharply. The time at which this sharp drop occurs corresponds to the point when particles are being absorbed into the tubular neighborhood. Consequently, if this time period is truncated through late initialization, particles will no longer be influenced by the score vector field as they naturally would. This makes it significantly more difficult for the particles to reach points on the data manifold, leading to an increase in the Wasserstein distance.

We will conduct ablation studies to validate this perspective.

Timing of Phase Transitions (Q2):
In Fig. 5(a), (b), and (c), the timing of the Wasserstein distance increase doesn't seem to be precisely aligned with the 0.99 threshold. For Fig. 5(a), this apparent misalignment is due to the separation between the Gaussian initialization region and the training data distribution. Appendix J.3 demonstrates how reducing the variance of the Gaussian initialization supports our hypothesis. For Fig. 5(b) and (c), the discrepancy arises from varying curvatures in the data distribution, as intended. These observations do not undermine our claim that the injectivity radius plays a significant role in the final stages of the generative process.

Statistical Visualization of Phase Transitions (Q3):
We can provide a summary of the Wasserstein distance values before and after the threshold to better illustrate the phase transitions. Please let us know if this would address your concern.

Experiments with Different Values of $R$ and $r$ (Q4):
We will include additional experiments varying the values of $R$ and $r$ in Section 5.3 to further validate our findings.

Reparameterized Observation of Diffusion Time (Q5):
If you are requesting a closer examination of smaller diffusion times, we can provide an expanded view of the range (e.g., $0$–$200$) in Fig. 5. Please let us know if this addresses your concern.

We appreciate your valuable suggestions and will incorporate them in the revised manuscript to strengthen our contributions.

Thank you for your detailed review and constructive feedback. Below, we address your concerns and suggestions point by point.

On the Strength:
Thank you for mentioning the strength. However, the results of our paper, more precisely as stated in Proposition 4.7, hold under conditions related to the "injectivity radius" and the time step. Proposition 4.7 establishes a sufficient condition for the score vector to point inward, as specified by inequality (10). It explains how larger injectivity radii $\epsilon_0$ facilitate satisfying this condition, whereas higher ambient dimension $d$ imposes more stringent constraints on the time step. We repeat that this is only a snippet of the assertion. Please read again carefully and investigate what is written in Proposition 4.7.

Phase Transition Definition and Tubular Neighborhood (Weakness 1):
You raise an important point regarding the definition of phase transitions and the special role of tubular neighborhoods. In our work, we view phase transitions as significant changes in the Wasserstein distance, supported by the theoretical framework of tubular neighborhoods. These neighborhoods are special because they are directly tied to the injectivity radius, which reflects the highest curvature regions of the data manifold. The disjoint arcs experiment (Fig. 7) demonstrates this, showing phase transitions corresponding to the highest curvature regions.

Regarding Fig. 5, we acknowledge that in some cases, transitions occur before particles fully enter the tubular neighborhood. This behavior, as discussed in Sec 5.2, arises from non-uniform curvature (Fig. 5(b), (c)). For cases like Fig. 5(a), the misalignment stems from the distance between the initial distributions in the forward and reverse processes. Additional results in Appendix J.3 support this analysis, showing alignment when the Gaussian noise variance is reduced.

Simplified Experimental Settings (Weakness 2):
We agree that our current settings are very simplified. Nevertheless, these choices aim to overcome limitations in prior work (e.g., Raya and Ambrogioni (2023)) and establish a clear baseline. The main problem to solve its limitation is that one does not simply analyze the free energy of the dynamics mathematically. In the case of ellipses, it is already a very difficult mathematical problem. We do think extending the analysis to more complex manifolds is indeed valuable future work. Computing free energy for more complex structures is mathematically challenging, and our approach seeks to address phase transitions using the tubular neighborhood framework as a practical alternative. Furthermore, this approach suggests a connection with the field of differential geometry and can be seen as a giant leap in interdisciplinary research.

Injectivity Radius in Diffusion Process (Weakness 3):

We agree that the significance of the injectivity radius requires further explanation:

• The score vector field plays a critical role in the behavior of diffusion models (Equation (2)).
• The behavior of the score vector field is determined by the injectivity radius (Proposition 4.7).

Given these facts alone, wouldn’t it be reasonable to say that the injectivity radius is a key factor in diffusion models? It is correct that curvature only accounts for partial behavior in diffusion models. However, the claim that the injectivity radius also captures only partial behavior is incorrect. Instead, it should be understood that the injectivity radius strongly influences the behavior at the final stages of the generation process (Proposition 4.7).

It seems that you are confusing curvature with injectivity radius. Curvature is a local quantity, while the injectivity radius is a global quantity. The guiding principle of statistics is to "always consider global quantities and understand them in global terms." If the statistical methods have a promise for mankind, it is better to consider global quantities rather than local ones.

While we acknowledge that spherical embeddings simplify the intrinsic geometry of datasets like MNIST and FMNIST, our results demonstrate consistent qualitative and quantitative behavior regarding phase transitions at the boundaries defined by the injectivity radius. This supports the applicability of tubular neighborhood theory even under these simplifications.

I appreciate the authors' response and get the point that

The guiding principle of statistics is to "always consider global quantities and understand them in global terms."

However, I still have some reservations about this as the injective radius, as a global quantity, is essentially decided by the most curved area of the manifold. That neighborhood are priori could occupy only a tiny portion of the manifold while the rest 99% has a much larger local injective radius. This makes the main result only a sufficient condition and not enough to capture the true phase transition happening along generic diffusion trajectories. I would like to keep my score.

Q1. Thank you for your detailed and insightful feedback. We appreciate the opportunity to clarify the key concepts related to late initialization, the tubular neighborhood, and the Wasserstein distance.

In our experiments, we measure the Wasserstein distance between the forward process's initial distribution $p_0(x)$ and the reverse process's final distribution $q_0(x)$. This distance evaluates the divergence between the training data and the generated data as a function of $T$, where $q_0(x)$ is obtained by initializing the reverse process at different time steps.

We made a hypothesis that the tubular neighborhood's radius plays a pivotal role in understanding critical phenomena. It delineates the region around the data manifold where particles are considered aligned with the data distribution. Late initialization impacts the reverse process by bypassing earlier transitions that would guide particles into this neighborhood, thus increasing the Wasserstein distance. Conversely, transitions occurring when particles are far from the neighborhood have minimal influence on the final distribution and can be ignored without negatively affecting performance.

We acknowledge that these connections were not sufficiently detailed in the manuscript and will revise the text to provide clearer explanations of these fundamental concepts. Thank you again for your critical comments, and we hope this response addresses your concerns.

Q2. We agree with your suggestion and will revise the phrasing to "following the definition" or a similar expression to improve clarity and precision. We appreciate your careful review and attention to detail.

Q3. Thank you for your suggestion. While we have already included a general description in lines 137--138 and provided the formal definition in Appendix C.10 for brevity in the original manuscript, we agree that moving the formal definition to the preliminaries could improve clarity.

Q4. Thank you for your comment. Theorem 3.7 is stated alongside Appendix D.1, and the numerical experiments using the pseudocode algorithm are also presented in Section 3.4 and Appendix F. On top of that, could you kindly clarify which part requires further elaboration? This would help us address your concerns more effectively.

Q5. Thank you for catching this typo. You are correct; $s$ should be $t$ in Equation (9). We will correct this in the revised manuscript. We appreciate your careful review and attention to detail.

Q6. We recognize the importance of providing more intuition behind the propositions in Section 4. The primary role of this section is to offer theoretical support for the experimental results presented in Section 5. We will improve the section accordingly.

To enhance clarity, we will expand on the implications of Propositions 4.2 and 4.7 through additional remarks (Remark 4.3, 4.8). Specifically:

• Proposition 4.2 demonstrates that the time derivative of $\Gamma(t)$ vanishes at $t=0$ and $t=\infty$, which aligns with the red curve's behavior in the Section 5 experiments. It can be understood that $\frac{d}{dt}\Gamma^{M(\epsilon)}(t)$ represents an integral of $\nabla u$ over the boundary. Here, the potential energy $\nabla u$ plays a key role in the diffusion dynamics (Raya and Ambrogioni, 2023). $\nabla u$ corresponds to the first term of Equation (2) in our paper.

• Proposition 4.7 provides a sufficient condition for the score vector to point inward, given by inequality (10). We will clarify how larger injectivity radii $\epsilon_0$ make this condition easier to satisfy, while high ambient dimensions $d$ impose stricter requirements on the time step. This is only a small snippet of the assertion. Please read again carefully and investigate what is written in Proposition 4.7.

We will revise Section 4 to better convey these intuitions and highlight their experimental relevance. Thank you for your constructive suggestion.

We have carefully revised the manuscript based on the feedback provided. Below, we outline the specific changes made to the relevant sections. These revisions primarily address clarity and presentation, while additional experiments are currently underway, as detailed below.

Section 4

• Notation Adjustment: All instances of s have been changed to t for consistency throughout the section.
• Remark 4.3: Revisions were made to clarify and improve the accuracy of the remark.
• Proposition 4.2: The notation f(t,x) was updated to f_t(x) for consistency with Eq. (3). A similar adjustment was made for g.

Section 5

• Figure Captions: The captions for Figures 4 and 5 have been revised to include detailed explanations of the blue and red lines, as well as descriptions of the axes.
• Dataset Removal: In Table 2, the datasets Ellipse (R=2, r=1) and Torus (R=2, r=1) have been removed due to space constraints. These datasets and their corresponding results have been moved to Appendix J.4 for reference.
• Notation Change in Tables: In Tables 2, 3, and 4, the notation for ρ_proportion has been updated from percentage format to ratio format for clarity.

Section 6

• Paragraph Adjustment: The final sentence of the first paragraph was commented out due to space limitations.

Appendix F

• Algorithm 1 Explanation: Additional explanations have been added to Algorithm 1 to enhance clarity and improve its understanding. We will continue to refine this explanation to ensure better readability and comprehension.

Ongoing Work

The changes made in this revision primarily consist of straightforward corrections and improvements to the manuscript. Regarding the additional experiments suggested during the review process, we are actively working on extending the mixed case analysis in Section 5.3. We aim to report the results of these extended experiments in a subsequent update.

In this update, we have incorporated additional experiments and added a new citation to extend the analysis in response to the reviewers' suggestions:

1. Following the suggestion from Reviewer 55op, we have added the citation: "Reconstruction and interpolation of manifolds I: The geometric Whitney problem," Fefferman et al., 2021, after Theorem 3.6 in Section 3.

2. As an extension of the Disjoint Arcs cases in Section 5.3, we conducted experiments with the following parameter settings:

   • (R, r) = (3, 1), where the injectivity radius is 1.
   • (R, r) = (3/2, 1/2), where the injectivity radius is 1/2.

The results of these experiments have been included in Appendix J.5.

3. For the elliptical case and Disjoint Arcs Case, we visualized the score vector fields at different time steps to provide further insights into their dynamics. These results have been included in Appendix K.

We hope these experiments will provide deeper insights into our study on the injectivity radius and its implications as discussed in the paper.