On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity

We would to thank Reviewer xahj for the very positive feeback, highlighting the compelling empirical analysis. Below, we provide answer to the specific questions.

The title and early claims argue against stochasticity as a driver of generalization, yet Section 4 appears to contradict this by showing that reducing stochasticity can improve performance.

We are confused by this sentence; in our opinion both the title and Section 4 consistently support the same message:

• The title argues against target stochasticity as a key driver of generalization
• Section 4 supports that target stochasticity does not bring generalization improvement compared to its more deterministic counterpart. Our findings tend to support that regressing against a deterministic target (EFM, Fig 5) yields generalization earlier during training. In particular, Section 4 shows that the more deterministic the target (i.e., the larger the number of samples $M$ to estimate the closed-form formula), the earlier the generalization occurs.

We would be grateful if the reviewer could elaborate on which part of the narrative appeared contradictory, so we can address it more explicitly in the revised manuscript.

1. In line 89, the notation “$z \sim p_0 \times \hat p_\mathrm{data}$” is ambiguous. Do you intend that $z$ is sampled from $\hat p_\mathrm{data}$ while an independent noise term is drawn from $p_0$ ?

What we meant here is that instead of choosing as conditioning variable $z = x^{(i)} \sim \hat p_\mathrm{data}$, an alternate choice (used eg by Albergo and Vanden Eijnden 2023, or Liu et al 2023) is to use $z = (x_0, x^{(i)})$ where $x_0$ is noise. This sentence has been removed for clarity.

2. Figure 5 is difficult to interpret: the gray curve is nearly invisible, and the reported gains from EFM appear marginal. I recommend reformatting the results—perhaps via a table or clearer color scheme—to improve readability.

Readability has been improved using larger figures. However, we would like to underscore that the goal of this figure is not to show that "Empirical flow matching (EFM) works better" than conditional flow matching. The core idea is to show that the estimator based on the stochastic target (vanilla conditional flow matching) and the estimator with a more deterministic target (empirical flow matching) yield very similar generalization performance. Hence, target stochasticity is not the driver of generalization.

In addition, as the reviewer observed the curves for $M=256$ and $M=1000$ are almost identical, which shows that above this threshold, using more points in EFM does not affect the performance. We have made this clearer in the discussion of the experiments.

3. Regarding the loss in Equation (7), can it be viewed as an interpolation between the conditional and unconditional flow matching losses? In particular, setting M=1 recovers the conditional loss, while M=the number of data seems to correspond to the unconditional loss.

Yes! This is exactly the idea!

• $M=1$ recovers the vanilla conditional flow matching loss, with a stochastic target.
• $M = \text{number of samples}$ recovers fully the flow matching loss, with a deterministic target, that would be very costly to compute in practice
• $1 < M < \text{number of samples}$ is a stochastic, but more deterministic target that is cheaper to compute in practice ($\approx 1$% overhead for $M=1000$). Please also refer to the discussion with Reviewer Byfq for additional details on the timings.

We would like to thank Reviewer ik8M for the very positive feedback and for highlighting the clarity and originality of the manuscript. Below, we answer the specific questions.

The analysis is done only over image modality and with U-Net architecture. Would be interesting to see if the same behavior happens for other modalities/architectures and how this affects the error in approximating the velocity.

We agree with the reviewer that our paper specifically targets image generation, as this is the main application for Flow Matching. In all the experiments, we used the reference Unet with attention and timestep embedding from the reference torchcfm library. We believe that we would obtain the same results with other architectures : for diffusion, [1] have shown that multiple architectures (DiT, U-ViT DDPM++, NCSN) end up learning the same velocity field (see their Fig. 1 and Fig 9)

[1] Towards a Mechanistic Explanation of Diffusion Model Generalization, ICML 2025, Niedoba, Matthew and Zwartsenberg, Berend and Murphy, Kevin and Wood, Frank

Can the authors extend the discussion on how the error in approximating the velocity relates to the dynamical regimes found in "Dynamical Regimes of Diffusion Models" ?

• The analysis in "Dynamical Regimes of Diffusion Models" [Biroli et al., 2024] focuses on the behavior of the closed-form score function in diffusion models, without considering how neural networks actually learn or approximate this function. Their main contribution is to identify three dynamical regimes—noisy, speciation, and collapse—that emerge when sampling is performed using the exact, analytical score in a Gaussian mixture setting. These regimes are derived from connections to classical phase transitions in statistical physics.
• In contrast, our work investigates how deep learning models approximate this ideal score function, and more importantly, how this failure to approximate the "optimal" score impact generalization. We empirically observe that model errors are concentrated around two critical timesteps (Figure 3), which may correspond to key dynamical transitions.
• Regarding the study of the non-stochasticity of the velocity field, we added experiments showing how the critical time for non-stochasticity (defined as a cosine similarity between $u^\text{cond}$ and $\hat u$ greater than 0.9) varies with both the number of training samples $n$ and the data dimension $d$. We observe a dependence in $\frac{\log n}{d}$, which is the same as the one obtained by Biroli for their 'collapse time' (defined as the time from which the diffusion trajectories are attracted by a single training point).

Drawing connections between the empirical failure modes we observe and the theoretical regimes identified by Biroli et al. is a promising direction for future work. Such a connection would require extending their theoretical framework to account for learning dynamics and approximation errors introduced by neural architectures.

We would like to thank Reviewer Byfq for the very positive feedback and highlighting the new perspectives on generalization provided by the paper. Below, we answer the specific concerns.

This paper does not give a clear mathematical definition for the "generalization" and build their frameworks purely on (somewhat vague) empirical results. For example, can the generalization ability be defined as the velocity matching error on the training set?

Mathematically defining and measuring the generalization of generative models is a challenging and still largely open problem. As per [1], there are three criteria: “quantifying to what extent images resemble those from the training set (fidelity), how well the generated samples span the full training distribution (diversity), and whether they are truly novel or are simply reproductions of training samples (memorization)”.

Standard evaluation metrics like FID have documented flaws [1,2]. For ourselves, we aimed to go beyond standard practice by:

• Using more expressive and discriminative embeddings (DINOv2) instead of the commonly used Inception features
• Measuring (approximate) Wasserstein distance between the representations of the generated set and the test set, rather than the training set, as it is usual done in generative modeling benchmarks
• Displaying a memorization metric that would detect a pure copy of the training set

We have extended the discussion in the Metrics paragraph L244 to clarify this discussion.

I agree with the authors that the additional samples can be efficiently tackled by vectorization and the same backpropagation depth. However, empirical evidence on computational time and memory should be provided, as EFM explicitly introduces additional costs.

• We emphasize that our main contribution is to challenge target stochasticity as the main driver of generalization: the main point of Section 4 is that the more deterministic EFM works at least as well as the stochastic CFM.
• However, given that EFM turns out to improve on CFM, its cost indeed becomes very interesting. To alleviate the non-linearity of GPU computing (parallelism may cause some discontinuities in terms of costs), we ran an exhaustive set of timing experiments, varying the batch size and the EFM sample size. To summarize the measurements (numbers are given for an NVIDIA L4 GPU, on CIFAR-10), denoting $b$ the batch size and $e$ the EFM sample size, the cost follows approximately $b \times ( 4.1\mathrm{ms} + e \times 0.04 \mathrm{μs})$. It can be also be seen as adding ~1% for every 1000 EFM samples. Or, for instance with a batch size of 256, 1.05 sec will be due to the 256-sample forward/backward, while the additional cost for EFM-1000 will be 10ms (around 1%) and for EFM-128, under 1.3 ms.

The experiments are not sufficient to prove the superiority of EFM, as different methods almost converge as the training proceeds. Moreover, to the best of my knowledge, quantititive results for flow matching methods can have large randomness.

We agree with the reviewer, and we emphasize that our claim is not that the EFM should become the new standard paradigm for training. We only mean to show that, when target stochasticity is removed from the problem (as in the EFM objective), models still generalize well, reaching as low FID as other training methods. This rules out target stochasticity as the driver for generalization. We added MNIST and will add FMNIST and tiny_imagenet by the end of the reviewers-authors discussion period, to strengthen this statement (see answer to reviewer MQbh).

In my opinion, the generalization ability generally leads to more evidently better test results than training results. However, in Figure 5, the training results and test results are almost the same. So can the authors explain this phenomenon?

In all our experiments, we indeed observe a strong correlation between FID scores on the training and test sets. This suggests that the models generalize well and that there is no significant overfitting in terms of FID. We believe this strong correlation between train and test FID may also reflect a limitation of the FID metric itself: FID does not penalize overfitting and does not detect subtle generalization gaps. This has been discussed in prior work (e.g., [2]).

While the close alignment between train and test FID may initially seem surprising, it is consistent with prior observations in generative model evaluation and reinforces the practice of using train FID as a proxy for test performance, though, as always, with some caution.

[1] Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., Liu, Z., Caterini, A.L., Taylor, E. and Loaiza-Ganem, G., 2023. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Neurips.

[2] Jiralerspong, M., Bose, J., Gemp, I., Qin, C., Bachrach, Y. and Gidel, G., 2023. Feature likelihood divergence: evaluating the generalization of generative models using samples. Neurips

We would like to thank Reviewer MQbh for the very positive feedback and for highlighting the clarity and care given the experiments. Below, we answer the specific questions.

1. While the paper highlights that generalization may arise from the “mismatch” between the neural network and the closed-form velocity field, it does not provide insight into the underlying inductive biases that make this mismatch beneficial. It is already known that the closed-form solution yields a perfectly fitted flow-matching model that memorizes and regurgitates the training data, making it ineffective for generalization. What would be more insightful is an exploration of which types of mismatch are beneficial for generalization, rather than simply observing that a mismatch exists.

There are two important ideas here. As the reviewer mentions, a closed-form solution exists that only generates training data. Thus, models that generalize fail to “match” this solution.

• Our first contribution is to discard “target stochasticity” suggested by Vastola (2025) as a cause for this mismatch: we show that this phenomenon is limited in realistic high-dimensional settings (Section 3.1), and that models still generalize well when target stochasticity is removed from the problem (Section 4).
• As a second contribution, we investigate "which part" of the mismatch is a key driver of generalization. Section 3.2 highlights "mismatches" at two specific times $t$: $t \approx 0.1$ and $t \approx 1$. Section 3.3 shows that the mismatch at time $t \approx 0.1$ is especially important for generalization: we conclude that failure to learn the optimal velocity at early time steps is the main driver of generalization.

2. Section 4 changes the training loss to effectively remove the stochasticity in the training objective. While I do agree that the empirical flow matching loss is “less stochastic,” it is not at all deterministic in any sense. It still relies on a Monte Carlo approximation and I do not believe this experiment alone fully obviates target stochasticity as an explanation for generalization (although I think the rest of the results complement this and the explanations are intuitive and convincing overall). Can you please add an experiment showing training scenarios where the batch hyper parameter in the EFM loss is also swept over?

We did not display the experiment with larger batches for the target, because our results showed that larger batches $M > 1000$ and beyond yield almost the same generalization performances, and the curves $M > 1000$ almost superpose. We have added the corresponding parameter sweep in the revised manuscript.

3. I would be interested in seeing similar experiments not just for CIFAR-10 and CelebA, but also for MNIST, FMNIST, and TinyImagenet to ensure that the conclusions are general enough.

We present below the results for the MNIST dataset. The conclusion is the same as for the other datasets: regressing against a more deterministic velocity field does not hurt generalization. On the contrary, generalization (i.e., lower test FID) appears earlier during training.

For this experiment, we used the Unet with attention and timestep embedding from torchcfm library, with the Adam optimizer and all the default parameters. We used a pretrained classifier with 99% accuracy on MNIST as a lower-dimensional embedding of size 128 to compute the FID between the test set and the generated set.

We will add the results for FMNIST and tiny-imagenet by the end of the author/reviewer discussion period.

Questions

• In L38, what is meant by the “deterministic regime” of the flow matching objective?: We call the "deterministic regime" the times for which the average velocity field $\hat u$ only correlates with a single of the $n$ conditional fields $u^\mathrm{cond}$, corresponding to the plot in Fig 1a, right.
• Adding x-axis in Figure 3 : this has been done