Metric Flow Matching for Smooth Interpolations on the Data Manifold

We thank the reviewer for their time and positive appraisal of our work. We are thrilled that the reviewer viewed our work to be “well-written, with great attention to detail” and that we “address an interesting problem” and is an “original contribution” that is a “useful addition to the generative modelling community”. We now provide responses to the main questions raised by the reviewer.

Weaknesses

1. It seems that the dataset $\mathcal{D}$ is the concatenation of samples from both $𝑝_0$ and $p_1$ but this is not clearly stated

Yes that is correct—we will make this more explicit, since we work in a “generalized” trajectory inference setup, we think of samples from $p\_0$ and $p\_1$ as samples from the same dynamical system evaluated at different times so that the manifold we refer to is indeed the one spanned by the trajectories.

2. There is no justification of the use of diagonal metrics except that it reduces computation cost. Surely there must be some downside? Is it possible to compare the performance to non-diagonal metrics in the 2d example?

That's a great question! Working with diagonal matrices, may potentially have some expressive power limitations in high dimensions, specifically for settings where capturing a precise underlying metric is crucial—note that in the applications considered in our submission it is hard to identify a ground-truth metric. In general though, there are theoretical reasons as to why our choice is not particularly limiting; for example, in the 2D case any metric is actually locally conformally flat (follows from the existence of isothermal coordinates). Accordingly, for 2D manifolds, one would not lose expressive power by actually taking the matrix $\mathbf{G}$ to be a multiple of the identity pointwise. Nonetheless, our framework is not constrained to diagonal metrics and more general cases are indeed possible. To further support this point, we refer to the additional experiments shared in the 1 page pdf and general rebuttal, where we showed how intermediate samples generated using MFM are much closer to the underlying lower dimensional manifold (a sphere) than the ones generated using CFM (see Table 4 and Figures 1,2,3). Crucially, MFM only leverages the diagonal LAND metric defined in the ambient space based on samples and is never given information about the sphere.

3. It's not clear how the OT is implemented. Is it a batchwise scheme? Does this introduce a bias? If not batchwise, is it scalable?

Our method relies on using mini-batch OT which has been a standard tool in the generative modeling / Machine Learning literature (see for example Tong et al., 2023 b,c). In this case we can easily trade off the cost of OT with the batch size and in practice this does not create a severe overhead in training our models. We find that OT helps improve training stability and leads to better empirical performance. We note however that MFM with independent coupling still surpasses CFM with independent coupling, as reported in our global 1 pg response where we compare CFM and MFM without OT (Table 1,2,3).

4. In tables 3 and 4, the MFM results are bolded. (..) I personally find it unscientific to report MFM values in bold and not others which could plausibly be lower than the MFM values

Thank you for your feedback. We agree with your observation and will remove bold numbers from Tables 3 and 4.

5. Line 289: I don't agree that the interpolations are better semantically. If there is a difference, it is too small to be able to make such a subjective claim

We again agree that this may be entering subjective territory a little, so we refine our statement and claims.

Questions

1. Line 125: "regular" means smooth in some way? Can you be more specific?

Yes, it does mean smooth in some way. To improve clarity, when citing the paper of Ambrosio et al we refer to their explicit statement, where assumptions are stated clearly.

2. Why is it preferable to use LAND for lower dimensions?

That's an excellent question. We empirically found LAND to be marginally better in lower dimensions, which we believe to be reasonable given that in this setting we are building the metric using all samples directly without any clustering being involved beforehand.

3. Am I correct that the LIDAR data is just an example of a 2d manifold embedded in 3d? (...)

Yes that is correct, as in we do not provide height information for the metric since we wanted to test MFM in a setting where the metric was built agnostic of the downstream application. To address your concerns, in the additional 1 page pdf we have provided more views which we will add in the appendix of the revised version to improve clarity.

The only limitation given is that the data must be embedded in Euclidean space. I am sure the authors can think of others, such as using a diagonal metric, and whatever trade-offs are involved in the OT scheme they use (see Weaknesses above)

Thank you for the feedback. We have now also added a line mentioning the trade-offs associated with the OT scheme. In light of our previous comment, we do not see any serious weakness with using diagonal metrics and in general the framework does not require one.

Final comment

We hope we have addressed the main concerns of the reviewer, and that the addition of experiments of MFM without OT and MFM on a sphere have further improved the strength of our submission. We hope that our answers enable the reviewer to continue endorsing our paper and potentially even upgrading the score if the reviewer deems it. We are also more than happy to engage in any further discussion.

We would like to thank the reviewer for their time and thoughtful review of our work, which gave us an opportunity to improve our work significantly. We are glad to hear that the reviewer found our work “well-written” and that it “proposes a solution that naturally connects the recently introduced flow matching method with a data-induced Riemannian metric” with “experiments that are well compared with recent studies”. We first refer to the general rebuttal, where we have provided additional experiments and visualizations (see the 1 page pdf). Below, we address the key clarification points and questions raised in the review one by one.

Weaknesses

I believe the trajectory inference problem with unpaired cross-sectional data has been well addressed in "Riemannian Metric Learning via Optimal Transport (ICLR 2023)." (..) Your approach and this work seem to have a fundamental difference, as the objectives used for learning the Riemannian metric are different. This difference should be clearly noted, emphasizing the advantages of your approach.

We appreciate the reviewer for pointing out this highly relevant work. We have already added a citation to Scarvelis & Solomon 2023 work in a revised version of the paper. We will further comment extensively on key differences: (i) the objectives being learned and (ii) the fact that we first optimize the paths and then match the vector field rather than regularizing directly the vector field (as in their Eq. (7)). Crucially, since MFM relies on flow matching, our framework does not require simulations during training when tackling the trajectory inference problem, which is an advantage compared to the work of Scarvelis and Solomon.

You have used L2 distance for ( c(x,y) ) in (12) due to computational issues. While this is understandable, I believe it could significantly impact the results. The two main factors affecting the resulting trajectory via flow matching are the Riemannian metric and the choice of coupling ( q ). (...) It appears necessary to include a more detailed discussion and address this point more thoroughly.

We appreciate the reviewer's concern. It is correct that the quality of the matching is given by both the choice of a coupling $q$—that is used to sample the boundary points $x_0$ and $x_1$—and the choice of the interpolants connecting such boundary points. First, we note that choosing a non-Euclidean cost for the OT objective is an interesting and non-trivial problem on its own. For arbitrary Riemannian metrics it is often unlikely to hope for a closed form expression to the metric-induced distance and thus OT cost. As a result, this requires us to simulate all pairs of matching to compute $c(x,y)$ which is computationally prohibitive from a practical standpoint. Furthermore, we argue an advantage of our framework is showing that even when we fix the cost to be Euclidean for the OT coupling, by learning data-dependent paths we can improve upon the Euclidean baseline. To further decouple the impact of the interpolants from the choice of the coupling, we conducted further experiments—found in our 1 pg global response PDF—comparing both CFM and MFM on the arch dataset and the single-cell ones, where we chose the coupling q to simply be the independent one. We agree with the reviewer that aspects of this discussion need further clarifications and we will include a larger section to explicitly outline our motivation and claims, along with the addition of the new experiments using independent coupling.

Minor comments

Thanks for sharing these works, we will add relevant citations in the revised version of our paper.

Questions

Both $g\_{\rm RBF}$ and $g\_{\rm LAND}$ capture data density and ensure that the interpolants lie within the data support. In principle, this algorithm appears to be designed to ensure that inferred trajectories lie within the data support. Are there other trajectory inference approaches where algorithms essentially do the same thing (i.e., identify trajectories that mostly lie within the data support and connect two data distributions)?

Regarding the choice of the metrics, we adopted those that seem to be the easiest/most efficient ones to show versatility of our framework and importantly reduce any computational overhead compared to the Euclidean FM baseline. Crucially though, we also wanted to highlight how our framework can, in principle, be applied beyond trajectory inference (as for the image interpolation problem), which is why we did not go overly-specific on the choice of the metric based on trajectories. However we believe this to be an interesting direction, i.e. how to choose task-dependent metrics to further improve the MFM framework. We hope this can be addressed in future work by us and the community more generally.

Final comment

We hope that our responses here in conjunction with the general rebuttal and the additional experiments help answer the great questions raised by the reviewer. We politely encourage the reviewer to continue asking more questions or if possible consider a fresher evaluation of our paper with a potential score upgrade.

We thank the reviewer for their detailed feedback and constructive comments, which gave us an opportunity to improve our work significantly. We are pleased to see that the reviewer found our work “very well written and easy to follow” and that our idea is “elegant” and that “the method works well without excessively requiring expert knowledge”. We now address key clarification questions raised by the reviewer.

Technical novelty

Beyond Eq. (4)(...) I see little "on top" of the work by Arvanitidis et al. (2016, 2021) and Tong et al. (2023b).

We value the reviewer’s comments regarding the technical contribution of MFM over prior works. We kindly refer to the general rebuttal for a detailed discussion on the main technical contributions of MFM, while here we provide a short summary.

First, we hope that drawing connections from metric learning—such as the works of Arvanitidis et al—and flow matching, is in itself an important contribution that can foster new works in this space bridging different communities further. We believe that showing the role played by the data geometry (not the ambient geometry) for generative modeling is an important research direction.

More crucially, in terms of technical contributions, we believe that the novelty is not just in the parameterization adopted in (4), but also, crucially, in the optimization in Eq. (6). In fact the objective in Eq. (6) is based on evaluating a data-dependent metric Dirichlet energy over the paths, once we sample the boundary points $x\_0,x\_1$ according to the coupling q. A key contribution in our work is that we can learn approximate geodesics using a simulation free learning objective which we find is quite useful for downstream generative modeling applications like trajectory inference. Moreover, we believe that optimizing using a velocity-induced regularization is an essential part of our framework, along with aligning the geodesic sampling with the joint distribution q. Crucially though, our framework is not bound to use a specific metric, and we simply adopted existing ones to showcase flexibility and ease of use. We hope this addresses the reviewer’s concerns, and clarifies the technical novelty of our paper.

Questions

Taking into account the weakness mentioned above regarding novelty: could the authors clarify their contributions beyond Eq. (4)?

We have addressed this point above when it comes to technical contributions.

I understand that in other domains, e.g. single-cell data, meaningfullness of trajectories might be more obvious to define, but I do not understand the claim about meaningfullness of trajectories between pictures of cats and dogs. (...)

We agree that meaningfulness of trajectories can be better evaluated over actual dynamical systems such as for single cell data. The experiments on unpaired image translation were meant to showcase that MFM can be on par, or in fact better, than the Euclidean baseline even on settings it was not mainly designed for. In particular, we sought to attach a visual correspondence to the learned interpolation from MFM. In contrast, one can consider another exotic interpolant which may traverse other classes—or entirely off the image manifold—in trying to go from cats->dogs. Note that to further try assessing the meaningfulness of interpolations we adopted the LPIPS metric.

Can a comparison with RFM be made in a setting where the underlying "true" appropriate metric is known? The Arch data set or some variant in hyperbolic geometry seem to be interesting cases for me.

This is a great suggestion! We address this with new experiments in our 1 pg PDF. We have extended the arch task on the 2D sphere embedded in $\mathbb{R}^3$. We have found that MFM not only improves significantly over the Euclidean baseline CFM (see Table 5 in the 1 page pdf), but crucially that the samples generated by MFM at intermediate times are much closer to the underlying sphere than the Euclidean counterpart (see Table 4 and Figures 1,2,3 in the 1 page pdf and comments in our general rebuttal). We emphasize that we manage to attain this without explicitly parameterizing the lower-dimensional space but simply relying on the LAND metric. Finally, we note that RFM uses the ground-truth geodesics of the standard metric on the sphere, meaning that in general how well RFM is able to solve trajectory inference problems on the manifold depends on how well the geodesics provided by the data-agnostic metric, resemble trajectories of the underlying system. Conversely, MFM directly learns from data and can approximately recover a curved, lower-dimensional manifold from the samples.

Minor comments

Why are there different citations for the manifold hypothesis?

This is just because we wanted to cite key papers in the main body of the paper (beyond the related work section) and the distinction (i.e. not citing all of them in both cases) is due to space constraints and formatting.

[l. 160] "boundary conditions": at this point I was defensively looking for some boundary conditions I might have missed, but later noticed the authors likely meant the endpoints and . Is that the case? Could this be clarified at this at this point when first mentioning "boundary conditions"?

Yes, by boundary conditions we mean that the paths recover $x\_0$ and $x\_1$ at times 0 and 1, respectively. We will add an extra sentence here to improve clarity.

Final comment

We thank the reviewer again for their review and detailed comments that helped strengthen the paper—particularly through the addition of MFM experiments on the sphere. We believe we have answered to the best of our ability all the questions here and in the general rebuttal. We hope this allows the reviewer to consider upgrading their score if they see fit. We are also more than happy to answer any further questions.

We would like to thank the reviewer for their detailed feedback and constructive comments, which gave us an opportunity to improve our work significantly. We are glad to hear that the main thrust of our work could be very “interesting” for higher dimensional ML applications. We kindly point the reviewer to our general rebuttal and 1 page pdf, which includes several new ablations and additional results. In this rebuttal, we address the main concerns of the reviewer: (i) Clarity on metric learning vs manifold learning; (ii) Novelty; (iii) Experiments using independent coupling.

Below the rebuttal, i.e. in the following comments, we tackle any standing point from the review.

Metric learning vs Manifold learning

The paper introduces their framework by discussing the "manifold hypothesis": that high-dim data lies in a low-dim manifold. However, the choice of metric does not induce a lower dimensional manifold.

The reviewer makes a pertinent observation that the choice of metric in MFM does not induce a lower dimensional manifold. To clarify why this is intended, given our data-points, we are interested in defining a notion of path between samples $x\_0$ and $x\_1$ that preserve the underlying geometry. There are now two equivalent ways to proceed: (i) One can find explicit coordinate representations of the underlying manifold and build the path through these coordinates (manifold learning); (ii) Define the path in the ambient space but change the geometry of the ambient space so that regions away from the samples (and hence the manifold) are highly penalized (metric learning). In practice we achieve the same result, which is having paths bending according to where the samples are. By leveraging the metric learning approach (ii), we do not need to prescribe a specific lower dimension or find a coordinate representation.

To support our point further, we have followed the suggestion of reviewer mgNS and run MFM on the arch task when the samples belong to the 2D sphere in $\mathbb{R}^3$. We have found that MFM not only improves significantly over the Euclidean baseline CFM, but crucially that the samples generated by MFM at intermediate times are much closer to the underlying sphere than the Euclidean counterpart (see Table 4 in the 1 page pdf and our general rebuttal). We emphasize that we manage to attain this without explicitly parameterizing the lower-dimensional space but simply relying on the LAND metric. We hope this provides you with strong empirical evidence as to why the metric approach can help us design flows that stay close to a lower-dimensional manifold even when we do not know it.

Novelty

The proposed algorithm is essentially the same as Wasserstein Lagrangian Flows and GSBM (…)

We value the reviewers opinion about the similarities between our proposed MFM and WLF and GSBM. We would like to gently push back against the reviewers assertion that these methods take a further step than MFM. In particular, we disagree with the assertion that WLF and GSBM learn an OT coupling induced by the choice of the metric, since no Riemannian metric (of any kind) is proposed or studied. As such, we argue WLF and GSBM cannot be considered generalizations of MFM despite their similarities.

We kindly refer to the general rebuttal where we have provided details on technical differences between MFM and WLF and GSBM.

One claim is that "MFM is simulation-free and stays relevant when geodesics lack closed form" when compared to riemannian flow matching. (...).

We appreciate the reviewers comment regarding the simulation free nature of the approximate geodesics learned in MFM. At present, there is no computational method that can find exact geodesics without simulations and we certainly do not hope to claim that MFM exactly solves these. In contrast, we believe that the strength of MFM lies in the fact that the problem of trajectory inference benefits from approximate geodesics that can easily be plugged in a conditional flow matching framework in a computationally efficient manner—i.e. they are simulation free. We further highlight that such approximate geodesics can be found thanks to our proposed objective Eq. 6, i.e.

where we minimize the Dirichlet energy of the path—whose optimum is in fact a geodesic. We point to the general rebuttal and to the 1 page pdf, where we showed how MFM manages to generate a flow whose intermediate samples are close to the lower-dimensional manifold by learning approximate geodesics of LAND through Eq. (6). We hope the reviewer may now agree that our empirical evidence enables us to claim both technical and empirical novelty of using learned (simulation-free) geodesics for trajectory inference.

Experiments using independent coupling

(...) The choice of interpolant is not independently studied with an independent coupling.

We thank the reviewer for this suggestion. In the general rebuttal we have provided additional experiments comparing CFM with independent coupling, i.e. I-CFM, and MFM with independent coupling, i.e. I-MFM, on the arch task and the single cell RNA datasets (in both 5D and 100D). We briefly summarize here the main takeaways:

• I-MFM generally surpasses I-CFM.
• We further stress that, differently from CFM, MFM also uses the coupling q in the first stage where it optimizes paths based on the metric, which justifies why using Optimal Transport for the coupling is even more beneficial for MFM than CFM.

We hope that in light of the new experiments, the reviewer can see that the benefits of MFM are not just due to the choice of the coupling, since I-MFM and OT-MFM both surpass their respective Euclidean counterparts I-CFM and OT-CFM.

When the optimal coupling is given, is there even a need to learn the flow?

That's an interesting suggestion! Unfortunately, single cell RNA is known to have a destructive generative process which means that different time marginals can contain varying numbers of unpaired observations. Operationally, this means there is no 1-1 correspondence between populations at different times. Consequently, evolving the particles along a path $x\_t$ may be an ill-posed problem and instead, we must assess the probability path $p\_t$. We achieve that by taking the pushforward of $p_0$ using the flow generated by the vector field $v_\theta$ learned using flow matching. We understand how this technical point may not have been clear given the complex nature of single cell data and now hope the reviewer agrees that their suggestion, while interesting, cannot be employed in this particular experiment.

I am a bit confused regarding the different dimensions (number of components used from PCA), since the initial motivation of this work is to learn the low-dimensional manifold. (...)

We appreciate the reviewer's concern. We politely point out that trajectory inference using PCA of single-cell data is not an innovation of our work but is a large subfield in its own right (see [2],[3]). Practitioners often resort to PCA as actual single cell data is both very high dimensional and noisy which provides a computationally intractable domain for learning trajectories prior to simulation free generative modeling. In this context we argue our reported benchmarks are standard in the literature. We also point again to our new experiments on the sphere (see general rebuttal), highlighting the ability of MFM to learn trajectories that stay closer to an unknown, underlying manifold.

Questions

How effective is the interpolant when no optimal coupling is used, especially for multimodal data distributions?

Please see our response/additional experiments on using independent coupling.

Does learning a full flow matching model outperform just using the L2 optimal coupling and using the interpolant to create x_t samples?

Please see our response above on why using just the interpolants for single cell RNA can be reductive due to the lack of a 1-1 correspondence between samples and the need to evaluate the time-evolution of the population density.

How does the framework work when you have multiple time snapshots? Is the metric time-dependent (by depending on different snapshots given time) or do you fit it to a single fixed-time snapshot?

Please see the reply to the first point of single cell experiment setup.

This part confuses me: to justify the metric, it is described that $p\_t$ "stays close" to a reference dataset D, however, the evaluation settings use K separate data distributions (at K different time values). How do you determine what D is?

Please see the reply to the first point of single cell experiment setup.

Given two disjoint distributions separated apart, does the metric still learn some non-trivial p_t between them (or does the metric degenerate to the Euclidean one when far away from data)?

This is an interesting question and we believe that given the setting you described, if the supports of the marginal distribution is fully separated, then there is no meaningful signal for the nonlinear correction $\varphi\_{t,\eta}$ in Eq. (4) and hence the path should remain linear, perhaps after adding a penalization over the norm of the weights of the MLP $\varphi\_{t,\eta}$.

Can this framework be adapted to learn a low dimensional manifold (rather than just a metric)?

Please see our response above on why MFM is not designed to learn a low dimensional manifold and why this is not an issue in practice (we point again to the new experiments of MFM on the sphere).

Final comment

We thank the reviewer again for their valuable feedback and great questions that enabled us to include new results that have strengthened our paper. We hope that our rebuttal addresses their questions and concerns—particularly in regard to how MFM does not need to prescribe a lower-dimensional space, as shown by the new experiments on the sphere, and the comparison with CFM using independent couplings. We kindly ask the reviewer to consider upgrading their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise.

References

[1]: “A geometric take on metric learning”, Hauberg et al., Advances in Neural Information Processing Systems, 2012.

[2]: “Best practices for single-cell analysis across modalities.”, Heumos, L., Schaar, A.C., Lance, C. et al. Nat Rev Genet (2023). https://doi.org/10.1038/s41576-023-00586-w

[3]:~ https://www.sc-best-practices.org/preprocessing_visualization/dimensionality_reduction.html

[4]: “Contrastive Learning for Unpaired Image-to-Image Translation” Park et al., Computer Vision-ECCV (2020)

In these comments we now address any standing question/concern.

For some of the cellular experiments, the manifold seems to actually be found by PCA

In single-cell data, current best practices suggest performing non-linear dimensionality reduction on the top 10-100 principle components. We follow this standard approach, learning a non-linear metric on the top 5/100 PCs, please see [2] and [3] below.

The paper uses very vague terminology. For instance, there is repeated mention of a "more meaningful matching"

We clarify that the term matching (introduced in Score Matching and Flow Matching [Song et. al 2019, Lipman et. al 2023]) refers approximating a target distribution $p_1$ with a learned one $p_{\theta}$ via a regression objective over vector fields—i.e. $|| v\_{\theta}(x\_t, t) - u\_t(x\_t | x\_0, x\_1)||\_2$. For trajectory inference tasks “more meaningful matching” refers to learning a matching that better respects the task description, meaning that the reconstructed trajectories replicate those generated by the underlying physical system.

We hope that our answer here addresses the reviewer's comment and we will update the main text to include these clarifications.

1. Lidar scan

It is unclear what the aim of this experiment is (...)

We thank the reviewer for their thoughtful suggestions. The primary purpose of our Lidar experiments is to simply contrast MFM to the Euclidean baseline (OT-CFM) and visually illustrate how the metric can affect the learned paths. While using more task-specific potentials—e.g. that account for the height information—as done in GSBM, is possible within our framework, we believe that this goes beyond the goal of this experiment which serves to highlight the benefit of incorporating data geometry without providing additional details from the downstream task.

To address the reviewer's concern about the faithfulness of $p\_t$ to the LiDAR data, in our general rebuttal and additional 1 page pdf, we also provide more views of the learned paths from OT-CFM and OT-MFM . The new visualizations clearly indicate that MFM paths 1.) do not intersect the manifold and 2.) bend closely around the manifold which highlights the geometric inductive bias.

2. The AFHQ experiment

Here the FID values are extremely high (...)

We acknowledge the reviewers' healthy skepticism regarding our reported FID values. We wish to highlight that other papers e.g. [4] have empirically reported FID values for interpolation of cats→dogs to be around 70. As such we believe that our values are acceptable.

Qualitatively looking at the samples, it doesn't seem clear which method has the better samples at t=1/2.(...)

We agree with the reviewer’s observation. Note however that in this application the goal is not much about having better samples at the intermediate times, but having samples at $t=1$ that are more similar to those at $t=0$ as measured by the LPIPS metric, which we have reported in Table 2. In general though, the main goal of this experiment was to showcase how MFM can be used along with or instead of CFM, even for settings outside the trajectory inference task which is what was mainly designed for.

Again, here I feel the role of the interpolant is heavily diminished when compared to the role of the optimal coupling.(...)

We thank the reviewer for their suggestions. As noted above and detailed in the general rebuttal, we have conducted experiments comparing CFM and MFM using independent coupling for both the Arch task and single cell RNA sequencing to validate that I-MFM surpasses I-CFM.

3. The single-cell experiment

Here the setup is given K time points, while one time point is missing. However, we know that by construction, the probability density at this missing time point only depends on the two closest time points(...)

This is an interesting question. In our framework, the metric $\mathbf{G}$ is constructed using two consecutive dataset marginals—based on a dataset $D\_{i,j}$, which is a concatenation of samples from these marginals (excluding $p\_{\text{out}}$). For instance, considering the scRNA dataset with densities $p\_0$, $p\_1$, $p\_2$, and $p\_3$, and excluding $p\_1$ as the left-out density, we use separate metrics $\mathbf{G}_{0,2}$ and $\mathbf{G}\_{2,3}$ for the pairs $\{p\_0, p\_2\}$ and $\{p\_2, p\_3\}$. In this regard, the definition of the metric already follows the procedure you suggested.

In contrast, a single time-dependent interpolant network $\varphi_{t,\eta}$ and a single vector field networks $v\_{t, \theta}$ are used for all time-steps ($t\_0$, $t\_2$ and $t\_3$) to ensure continuity of trajectories across different times. This also ensures that we follow standard procedures adopted by the baselines reported in Tables 3 and 4.

Issue with the choice of coupling (...)

We have addressed this point above and in the general rebuttal. Once again, we highlight that I-MFM is better than I-CFM.

Unfortunately, the authors did not seem to understand my concerns about clarity and the paper's claims. I want to clarify that the only concern that was addressed by the rebuttal was that the choice of metric has advantages regardless of the choice of coupling used during training.

Metric:

"to the best of our knowledge, existing works do not use parametric metrics" --> To be clear, by parametric (in contrast to non-parametric) I simply meant that the metrics do not depend on a large data set. But I believe here the authors are using parametric as a synonym for learned; however, even then, this claim is certainly false. There has been many works on learning parametric metrics over data sets / point clouds [1,2], and many works on using latent spaces for transport problems where the metric is induced by the autoencoder, such as the DSBM and GSBM methods [3,4]. Relatedly, it may be worth noting that the authors did not address my concern on reproducing FID: DSBM reports 14.16 and GSBM reports 12.39 FID. Both are significantly better than the values reported by this paper (37-41), and this is especially concerning because existing works have open source implementations available. Instead, it seems the authors have deliberately chosen not to compare against the different choices of potentials in these works.

"in favor of potentials (that are generally not defined)" --> I am not sure what is meant by not defined here, as I understand these prior works work for any potential. Since the potentials can be arbitrarily defined, it fully includes the Lagrangian point of view of Riemannian geodesics. As mentioned in Appendix C.1, one can always convert from the cost defined using a Riemannian metric to the regular Euclidean metric plus a potential.

"do not adopt a Riemannian formalism to assess the velocity of the path" --> I believe the authors are describing how to extract a Riemannian manifold from a data set. There is significant work in this area (see the literature on constructing Riemannian metrics from point clouds e.g. [1, 2]) and is often used for LiDAR. Actually, it is worth highlighting that my concern regarding the LiDAR experiment has also not been addressed: compared to the GSBM experimental result which show two modes of trajectories [4; Figure 4], the results from this paper only show one mode: why is this? For me, I am not convinced that this LAND metric is as good as the choices of potential compared to existing works.

Cost and Complexity: Here is a simple question: Given the LAND or RBF metric definition, do you have access to all geodesics in closed form? If so, then I agree the method would be simulation-free and very computationally efficient; however, from my understanding, the answer is no. And the proposed workaround is to actually fit a parametric approximation to the geodesics. This, as the authors also agree, have significant computational overhead and can increase costs to double the amount for CFM due to this extra step of needing to approximate geodesics with neural networks.

Manifold hypothesis: Taking wikipedia as a proper "definition", the manifold hypothesis, if it were to be true, would imply at least two things:

1. "Machine learning models only have to fit relatively simple, low-dimensional, highly structured subspaces within their potential input space (latent manifolds)."

This, as also reflected by the rebuttal, does not hold for the proposed method. The proposed method works in the original data space.

2. "Within one of these manifolds, it’s always possible to interpolate between two inputs, that is to say, morph one into another via a continuous path along which all points fall on the manifold."

As noted in the review, and also agreed upon by the rebuttal response, the intermediate samples (see Figure 3 for images, and the rebuttal pdf for sphere and LiDAR) certainly do not seem to lie on the data manifold. This is a subtle point that I raised which I believe the authors have not understood. If one were to actually define a manifold (along with a tangent plane and a corresponding metric on that plane), then interpolations will never leave the manifold, no matter how poorly the model/distribution is fit. This is the key idea behind the literature on constructing Riemannian manifolds as described above, where we can fully construct a proper manifold and not just a metric in the ambient space. However, the proposed approach does not learn a subspace and due to this, we see that the points are some distance away from the manifold (shown for both sphere and LiDAR visualizations in the rebuttal pdf).

[1] "Neural FIM for learning Fisher information metrics from point cloud data" Fasina et al. 2023.

[2] "Approximating the Riemannian Metric from Point Clouds via Manifold Moving Least Squares" Sober et al 2020.

[3] "Diffusion Schrödinger Bridge Matching" Shi et al 2023.

[4] "Generalized Schrödinger Bridge Matching" Liu et al 2023.