Thanks for detailed feedback and for recognizing the importance of addressing ambiguity in the marginal velocity field, and our strong results.

1. Paper's unique contributions

We study a method for capturing multi-modal velocity vector fields. We show that incorporating an unobserved continuous latent variable z via a variational formulation (akin to a VAE) enables the velocity model to learn a multi-modal vector field. Experiments across diverse data and models show that our method outperforms classic approaches.

2. How many z used for large-scale experiments. Averaging

To remain fair and as shown in Algorithm 2, we sample a single $z$ per data point and keep it fixed throughout integration. No averaging is performed on $z$ or predicted velocity.

3. Impact of KL weight

We summarize the impact of the KL weight $\lambda$ based on our experimental findings:

1. The model successfully captures velocity ambiguity and predicts crossing flows when $\lambda$ is in a reasonable range (in [0.1,10.0]).
2. When $\lambda$ is large (e.g., 100.0), the latent model is forced to equal a standard Gaussian. Hence, the latent z contains minimal useful information. Hence, the velocity network behaves similar to one obtained using classic rectified flow. This is also apparent in the loss: the KL loss diminishes, and the velocity reconstruction loss is comparable to the baseline loss. The resulting flow cannot capture ambiguity.
3. When $\lambda$ is small (e.g., 0.01), the model can exploit excessive information from the latent. This leads to a very low velocity reconstruction loss but a very high KL loss. The resulting flow appears as straight lines, but the endpoint distributions do not match the target data due to the mismatch between the predicted posterior and prior.

In our experiments, we didn't tune the KL regularization weight much, but instead scaled the KL loss with the dimension of the latent variable. E.g., for ImageNet SiT experiments, we directly used the KL weight that we employed for CIFAR-10.

4. Implementation of $q_\phi$and $v_\theta$

As stated in Sec 4.4 (L342 - 343), $q_\phi$ and $v_\theta$ share a similar structure but are separate nets. We will clarify this to avoid confusion. Regarding the increase in parameters, as described in Sec 3.3 and Sec 4 (L262, 348, 372 right column), during inference, $q_\phi$ is not used. Instead, we sample the latent variable from a prior. The only increase in parameters comes from the two MLP layers to fuse the latent z in $v_\theta$. This design ensures that our velocity network remains comparable in size to the baseline, with less than a 2% parameter increase for CIFAR-10 and less than a 0.3% increase for ImageNet.

5. Code Snippet

We will release code and models. Note, we provided more implementation details in App D. We are happy to address further questions during the rebuttal phase.

6. More baselines: consistency models, shortcut models, HRF

Consistency model: A detailed comparison to consistency models, particularly distillation models, is included in App B. We used the recently developed consistency flow matching [1]. It improves upon consistency models [2] and is more closely related to flow matching. We summarized the results in App C.1. Our key findings:

• The consistency flow matching model performs well at low function evaluation regimes (i.e., with NFEs of 2 or 5).
• Its performance degrades as NFEs increase.
• Its best performance across all NFEs remains below classic rectified flow matching and our variational rectified flow matching.

We also highlight an exciting future research direction: combining variational flow matching with consistency models, which could further enhance results.

Shortcut model: We evaluate the Shortcut Model (XL), trained for 800k iterations, using the FID score and following the same evaluation protocol used in Tab 2. Our results show that our method consistently outperforms it.

Hierarchical Rectified Flow (HRF): This concurrent work also aims to model multi-modal velocity and acceleration fields but uses a hierarchical rectified flow. Their method requires multiple integrations during inference, making it slower than our approach. Also, HRF does not support semantic disentangling of flows, as demonstrated in our Fig 6 and 7 for MNIST and CIFAR-10.

[1] Yang, L. et al. (2024) Consistency flow matching.

[2] Song, Y. et al. (2023) Consistency models.

7. || should be used instead of | in KL

Thanks for spotting this typo, we'll fix.

8. resources for SiT-XL on ImageNet 256

We used 8 H100 GPUs and trained the model for about 3.5 days.

Thanks for feedback and for highlighting our theoretical contributions and strong results across datasets and models.

1. Quantify the average curvature of 2D trajectories

We calculated the curvature for 2D data results (Sec 4.2) and find significantly lower curvature for our method:

2: How to obtain desired digits/brightness via $z$. Latent interpolation would strengthen the claim

Great suggestion. We conducted interpolation experiments and summarize findings verbally as images can't be uploaded. For MNIST, interpolating latents leads to smoothly transitioning digits (e.g., 1 → 7 → 8 → 3 → 2 → 0). Note, Fig 6 illustrates that each digit corresponds to a specific latent $z$. Intermediate digits emerge naturally when interpolating. For CIFAR-10, we observe analogous effects—interpolating between two latents leads to smooth transitions in brightness and color patterns.

3: Marginal data distribution preservation lacks a proof

The proof in App A refers to results established by Liu et al. to provide adequate credit. This may require readers to reconstruct parts of the proof using work of Liu et al. To make the paper self-contained, we will expand it to add the missing steps (derivation of $d/dt E[h(X_t)]$ and equivalence of $0 = E_Z \int(...))$.

4. Formulations of OT-FM and I-CFM are almost equivalent. Redundant comparison with I-CFM

OT-FM/I-CFM differ in $x_t$ and the conditional vector field $u_t$. Specifically, OT-FM defines $x_t$ as $\mathcal{N}(t x_1, 1-(1-\sigma) t)$ and $u_t$ is $\frac{x_1-(1-\sigma)x_t}{1-(1-\sigma)t}$. I-CFM defines $x_t$ as $\mathcal{N}(t x_1+(1-t)x_0, \sigma)$ and $u_t$ as $(x_1-x_0)$. SiT uses the rectified flow objective, which is equivalent to I-CFM. We'll follow the suggestion to remove OT-FM for consistency.

5. Figure 12 of MNIST FID score should be table

We'll convert Fig 12 into a table.

6. Discussions of related work

Thanks for highlighting these works. We'll add a full discussion. Below is a brief summary:

1. Denoising Diffusion GANs replace the Gaussian model in the denoising step with a multimodal distribution. Unlike our method, a conditional GAN with separate discriminator models the distribution. But GANs face mode collapse and stability issues. In contrast, our method uses rectified flow matching, preserving the maximum likelihood benefits.
2. Score-based Generative Modeling uses a VAE to map raw data $x_0$ into latent space $z_0$, with the VAE jointly trained with score-based generative modeling (SGM). Unlike our approach, SGM still faces ambiguity issues due to its use of a uni-modal Gaussian distribution.
3. Stable Target Field notes that the posterior distribution is multi-modal. To model this distribution, the paper reduces training target variance using a reference batch. In contrast, our method directly models this multi-modal posterior via a recognition model.

7. Improvements between SiT-XL and V-SiT-XL diminish as training continues

A diminishing gap is expected as absolute values decrease, a trend also presented in Fig 2 of the SiT paper (comparing the gap of DiT v.s. SiT). However, the relative improvement remains strong. We extend training to 1200k steps and report both absolute and percentage improvements. We observe the percentage improvements increase with more training iterations.

8. Improvements are small, not comparable to distillation

We respectfully disagree. Our improvement is solid, while strictly following the open-source SiT training. Results are consistent with the SiT-to-DiT improvement, demonstrating a comparable level of progress (19.5 for DiT-XL, 17.2 for SiT-XL, and 14.6 for V-SiT-XL at 400k steps, as shown in Tab 2).

Also note, our key contribution is to model the velocity distribution. We find this to consistently improve evaluation metrics across all NFEs. While distillation methods may show improvements for low NFEs, our method achieves better results across both low and high NFEs.

Additionally, as discussed in App B, V-RFM focuses on single-stage training to capture a multi-modal velocity distribution from “ground-truth” data without leveraging pre-trained models. Exploring distillation for V-RFM is an exciting avenue for future research, particularly when the interest is to improve results for low NFEs.

9. Replace "data-domain-time-domain"

Great suggestion. We'll revise.

10. ImageNet performance below 250 NFEs

Fig 8 shows those results, revealing a consistent boost, further highlighting our method's effectiveness.

Thanks a lot for your detailed feedback and for recognizing our well-written paper, extensive experiments, and comprehensive evaluation of performance. We also appreciate the acknowledgment of V-RFM’s effectiveness in addressing velocity ambiguity and its superior performance. Below, we address questions:

1. The method requires additional parameters to compute the latent representation during training, increasing computational complexity.

Yes, extra computation is used during training. It is essential to extract latent information within the velocity network, ultimately enhancing generation quality and expressiveness. As discussed in Sec 3.3 and 4, the latent encoding network is not used during inference—we directly sample $z$ from the prior distribution. Furthermore, the increase in parameters is minimal (e.g., less than 0.3% on ImageNet), and the impact on inference speed is negligible while delivering superior results. The ablation study on the size of the posterior model summarized in App Tab 5 shows that performance remains consistent across variations, demonstrating the robustness and flexibility of our approach. This allows users to balance training efficiency and runtime quality based on their computational constraints.

2. While the proposed method outperforms baseline, it lags behind the current SOTA.

The primary contribution of our work is not in surpassing the current state-of-the-art, but rather in introducing a methodological innovation, i.e., capturing the multimodal velocity field, which offers new avenues for improvement in the field. While our method lags behind the state-of-the-art, it can be combined with innovations put forth by current SOTA SiT-XL/2 + MG [1] and SiT-XL/2 + REPA [2], both built upon the SiT framework. As detailed in our experiments (Sec 4.5, L364-367 right column), we strictly followed the original training recipe from the open-source SiT repository and replicated the process outlined in the SiT paper for a fair comparison. Our experimental results provide empirical evidence of the effectiveness of our approach.

[1] Tang, Z. et al. (2025). Diffusion Models without Classifier-free Guidance.

[2] Yu, S. et al. (2024). Representation alignment for generation: Training diffusion transformers is easier than you think.

3. A definitive ground truth mapping may not exist.

A definitive “ground-truth” does not exist indeed. The (“ground-truth”) velocities form a distribution for every $(x_t, t)$-location. To see this we independently sample from both the source distribution and the target data distribution, calculate the rectified flow interpolants for each pair, and visualize them in Fig 1 and the velocity distribution in Fig 3(a). Our method models this velocity distribution at every $(x_t, t)$-location, while standard rectified flow matching cannot capture it. In the abstract and the main paper, we use quotation marks around “ground-truth” to emphasize this distinction. We will correct any missing instances in the final version.

4. Provide a rigorous theoretical proof to substantiate the claim that the proposed method resolves vector ambiguity.

In Sec 3, we show that the proposed approach leads to a mixture model for the velocity distribution (L179-180). A mixture model is theoretically capable of capturing multi-modality. Following classic expectation maximization or variational inference, we introduce the recognition model, derive the lower bound of the marginal likelihood for an individual data point (L189-192), and present the variational flow matching objective (L197-199). To further substantiate our approach, in App A, we show how to prove that the distribution learned by the variational objective preserves the marginal data distribution. Empirically, on 1D data we visualize the learned velocity distribution in Fig 3, showing that the method indeed learns the velocity ambiguity. Further, during training on high-dimensional data, we observe that our method achieves better velocity reconstruction losses (App Fig 15) compared to standard rectified flow, indicating that the predicted velocities more accurately approximate the “ground-truth” velocities.

5. How does the choice of latent variable affect the generative path.

We studied the role of $z$ for MNIST data (Fig 6) and CIFAR-10 data (Fig 7). As noted in Sec 4.3 and 4.4, we observe clear patterns in the generated samples based on $z$. Specifically, images conditioned on the same latent $z$ exhibit consistent color patterns, while images at the same grid location show similar content. These observations validate the effectiveness of the latent variable $z$ in influencing and controlling the generated samples.

Thanks a lot for your detailed feedback and for recognizing V-RFM's novelty and promise as a generative model. See below for answers to your questions:

1. Training stability of V-RFM. VAE training is not as stable as RFM because it may encounter the posterior collapse problem.

Great question. During our studies we observed that stability can be controlled by the architecture. Specifically, as stated in L353-357, bottleneck sum, which fuses the latent $z$ with activations of the velocity network, and adaptive normalization, which explicitly scales and offsets the latent $z$ at multiple layers of the velocity network, are effective at ensuring that the latent variable $z$ sampled from the posterior is not ignored. As shown in App Fig 15, the reconstruction losses of V-RFM remain consistently lower than the baseline, demonstrating that the latent variable contributes meaningfully to reducing the reconstruction loss. Furthermore, Fig 6 and 7 confirm that modifying the latent $z$ alters the predicted image, further verifying that our latent representation remains informative and utilized throughout training. Lastly, we conducted an ablation study on the size of the posterior model, summarized in App Tab 5. The results show that performance remains consistent across variations, indicating that even with a very small encoder (6.7% of its original size), the latent information remains informative and helps achieve competitive FID scores.

2. More baselines in Tables 1 and 2, such as flow matching and 1/2/3-Rectified Flow.

Note, the OT-FM/I-CFM baselines in Tab 1 and the SiT baseline in Tab 2 employ the flow matching objective with differences in the parameterization of $x_t$ and the conditional vector field $u_t$. Specifically, OT-FM parameterizes $x_t$ as $\mathcal{N}(t x_1, 1-(1-\sigma) t)$, while I-CFM defines it as $\mathcal{N}(t x_1+(1-t)x_0, \sigma)$. The conditional vector field $u_t$ is $\frac{x_1-(1-\sigma)x_t}{1-(1-\sigma)t}$ for OT-FM, while it is simply $(x_1-x_0)$ for I-CFM. We also note that 1-Rectified Flow is equivalent to I-CFM.

We have added a comparison with 2/3-Rectified Flow via Reflow. We find that while strong FID scores in the low-NFE regime are achieved, it does so at the cost of limiting peak performance at high NFE. We emphasize that Reflow is a supplementary technique applied on top of Rectified Flow Matching (RFM)—primarily aimed at fast sampling rather than improved sample quality. It also requires $N$ times longer training and a significantly larger fine-tuning dataset, where $N$ denotes the number of Reflow rounds. These differences make a direct comparison with our V-RFM less fair. Hence, RFM without Reflow is a more appropriate baseline. Additionally, Reflow can be applied to our method as well, potentially improving results at the cost of increased training overhead. We will clarify these points in the final version and include additional qualitative comparisons for a more precise evaluation.