Elucidating Flow Matching ODE Dynamics via Data Geometry and Denoisers

We thank the reviewer for the positive feedback and constructive comments. Thank you for appreciating the convergence guarantees and insights into the memorization behavior of flow and diffusion-based generative models. We will fix the missing parenthesis in the definition of the medial axis to make it clearer in the revised manuscript.

We would also like to point out that we have further strengthened our theoretical results with new experiments on FFHQ to show the utility of our theoretical findings (in addition to results we already have in the appendix); see response to Reviewer m3yf, as well as discussions to potential practical implications as in response to Reviewer zqMS. We will clarify these points in the revision. We believe that incorporating those changes will further strengthen our paper.

We appreciate the reviewer’s positive feedback and recognition of our novel theoretical contributions on the FM ODE convergence under weak assumptions, per-sample trajectory evolution patterns, and connections to memorization. We believe that the per-sample level analysis is important as it could inform data geometry-based sample step scheduling or steering strategies. Below, we respond to the comments/questions:

Cluster assumption and prior noise design: We agree that the well-separated local cluster assumption may not hold for all datasets. Our analysis suggests similar attracting and absorbing behavior could happen for locally dense regions. For example, we can prove a particular case where data distributions obtained from well-separated clusters convolved with Gaussian noise. Specifically, Lemma C.11 identifies FM ODE trajectories with additionally convoluted Gaussian noise with early-stopped FM ODE trajectories w.r.t. the original distribution (up to parameterization), which can help extend current Proposition 4.4 to this more general setting.

In practice, our analysis suggests that prior noise with a bias towards the cluster center could effectively control the sampling outcome to reflect the desired cluster characteristics. This could hold, in general, dense regions of the data distribution, as demonstrated in our new FFHQ experiments below.

Cluster experiments on FFHQ: Following your suggestion, we conducted new experiments on the FFHQ dataset, which is similar to CelebA, with a pre-trained EDM model (Figures in PDF: https://anonymous.4open.science/r/ICML-7924/FFHQ_exp.pdf). Figure 1's t-SNE plot shows that images with similar illumination tend to cluster together, especially at brightness extremes, as RGB encoding makes brightness differences strongly influence distances in pixel space. This creates distinct dense regions for the darkest and lightest images.

We performed sampling using a pre-trained EDM with three initialization strategies: random, near-light, and near-dark regions. All samples used standard EDM sampling (18 steps from noise 80 to 0.002), varying only initialization noise. The results in Figure 2 show that when initialized randomly, samples exhibit various illumination levels. When trajectories start near light regions, samples consistently display light characteristics, while dark initialization produces dark samples. These validate our theoretical findings on cluster absorption even where clustering is based on subtle, continuous features rather than discrete classes.

Non-random pairing and non-Euclidean case: We suspect convergence still holds for non-random pairing, though general dynamics might differ significantly. For non-random pairings (as in rectified flow), the posterior distribution $p(x|x_t)$ is a Dirac delta, and the posterior mean is determined by the deterministic pairing. Hence, we suspect that the convergence in terminal time still holds in this case. But the ODE dynamics in other stages are different: for example, paths may straighten after rectification (due to convergence to optimal coupling), then there will be no travel to the data mean in the initial stage. The non-Euclidean case can be more challenging, as the transition kernels in those cases might be more complicated than Gaussian kernels. These topics are interesting and worth exploring in the future, and we will provide more discussions in the revision. We want to reiterate that our analysis of random pairing in Euclidean space is broadly applicable, as most generative models fit this setting.

Consistency model and equivariance We agree that it would be beneficial to better connect our convergence result to consistency models and equivariance.

The consistency model aims to distill the entire flow matching trajectory (from noise to data) into a single map, which necessarily requires the FM ODE to converge. Our work formally validates this convergence, ensuring consistency models are mathematically well-defined. We will add a discussion on this connection following our main convergence results.

The equivariance result naturally extends from the convergence of FM ODE trajectories. Additionally, it provides another perspective on how data geometry affects the ODE trajectories: a similarity transformation results in some deterministic and explicit transformation of FM ODE trajectories.

$C_{\epsilon}^{S}<1$ in Proposition 4.4: This condition ensures the formula is well-defined. It could fail when the weight $a_S$ on a cluster is large, but this is not problematic. In fact, when $C_{\epsilon}^{S}\geq 1$, the cluster exhibits a stronger attracting force and $\sigma_0(S, \epsilon)$ equals infinity, meaning that Proposition 4.4 would apply to any initialization time not just after time $\sigma_0(S, \epsilon)$. We will clarify this in the revision.

We will incorporate these changes in the revision. Thank you for your feedback; we hope the revision addresses your concerns.

We thank the reviewer for the constructive feedback and for recognizing the theoretical importance of our work on the convergence of FM ODE dynamics, the analyses connecting data geometry with FM model trajectories, and the memorization phenomenon. Below, we address the main comments/questions:

Experimental validation: We conducted experiments on synthetic data and CIFAR-10, presented in the Supplement due to space constraints. Specifically, we demonstrated the emergence of the three stages of ODE evolution on synthetic data (Figures 7- 10) and the convergence to data mean (Figure 11) and memorization phenomenon on CIFAR-10 (Figures 13, 14), aligning with our theoretical findings in sections 4 and 5. Following Reviewer m3yf's suggestion, we have now also conducted experiments on the human face dataset (FFHQ). Please see the response to Reviewer m3yf for more details.

Assumption on positive reach: First, we note that this assumption is only used for our convergence results (Thm 5.3, 5.4) and equivariance (Prop. 5.7), not elsewhere in the paper. Also, the positive reach assumption is common in manifold learning literature (e.g., Fefferman et al., 2016) and is not overly restrictive. Specifically, the medial axis $\Sigma_{\Omega}$ of a manifold $\Omega$ is the set of points in the ambient space having more than one closest point on $\Omega$. The local feature size $lfs(x)$ at a point $x \in \Omega$ is the distance from $x$ to the medial axis $\Sigma_{\Omega}$. The reach of $\Omega$ is defined as $\min_{x\in \Omega} lfs(x)$. The local feature size of any submanifold embedded in Euclidean space is positive everywhere. Consequently, all compact submanifolds have positive reach, which includes most real-world datasets under the manifold hypothesis (e.g., image patches, audio).

Moreover, our core analysis (e.g., Theorem 4.1) is local, meaning the positive reach assumption could potentially be relaxed to only require positive local feature size. This relaxation would extend our results to an even wider class of data distributions, including singular manifolds (e.g., sheets of manifolds glued together with singularities).

Fefferman et al. "Testing the manifold hypothesis." JAMS 29.4 (2016): 983-1049.

Practical implications of our theory: We believe that understanding the behavior of per-sample ODE trajectory is critical, as that is the trajectory followed during the inference stage. Our theoretical analysis has several direct practical implications for FM model design and training:

1. Memorization mitigation strategies: Our analysis in Section 5.2 reveals how the terminal stage of ODE evolution influences memorization phenomena, suggesting potential regularization approaches based on the denoiser near the terminal time–e.g. regularize the Jacobian of denoiser.
2. Latent space design principles: Our analysis reveals how data geometry directly shapes ODE trajectories, with important implications for latent space design. By leveraging the data's clustering structure, one can design more effective latent representations that facilitate both efficient sampling and precise feature formation. Our equivariance result suggests that preserving key equivariance properties can lead to more robust, generalizable, and interpretable models.
3. Theoretical foundation for one-step distillation method: Our convergence results provide rigorous mathematical justification for one-step distillation approaches like consistency models. These methods rely on the assumption that FM ODEs converge to stable, well-defined mappings, which our results formally validate.
4. Optimized sampling strategies: Our characterization of the three-stage ODE evolution suggests adaptive sampling approaches that allocate fewer integration points to the initial and final stages while concentrating computational resources on the intermediate feature formation stage. This provides theoretical grounding for empirically successful non-uniform time discretization schemes and explains why the practice of starting at moderately large noise levels (e.g., 80 in EDM) is successful, although infinite initial noise is required theoretically.

Other comments: We will improve the paper's clarity in the revision to make the notation and concepts more accessible. For example, the hashtag in the medial axis definition is a shorthand notation for the cardinality of a set, and we will explain this explicitly in our future revision. Regarding the Jacobian of the denoiser, we will update its discussion and include the suggested references. For its relation to the data geometry, as the denoiser converges to projection onto the data manifold, the rank of the Jacobian of the denoiser can reflect the local dimensionality of the data, which could help indicate if the trained model is memorization- when the rank is very low.

We will incorporate these changes in the revision. Thank you for your feedback and we hope the revision will address your concerns.