Consistency Flow Matching: Defining Straight Flows with Velocity Consistency

Q3: In Theorem 1, the authors consider no exponential moving average, and the ground truth velocity field is assumed to be known. I recommend discussing the impact of these assumptions on practical applications.

A3: Thank you for raising this important point. We address the impact of these assumptions on practical applications as follows:

In practical experiments, we found that training with $\theta^-$ replaced with $stopgrad(\theta)$ leads to a slight improvement in FID. Therefore, considering no exponential moving average does not significantly affect the validity of Theorem 1.

In the updated Appendix A.6, highlighted in blue, we provide Theorem 3, which shows that training Consistency-FM using the conditional trajectory $x_t, x_{t+\Delta t}$ is equivalent to training with a perfectly pre-trained FM trajectory that estimates the ground truth velocity field. The equation below is from the updated Theorem 3:

$$E|| f\_\theta(t,x\_{t}) - f\_{\theta^-}(t+\Delta t,x\_{t+\Delta t}^{FM})||\_2^2 = E|| f\_\theta(t,x\_{t}) - f\_{\theta^-}(t+\Delta t,x\_{t+\Delta t})||\_2^2 + o(\Delta t)$$

where $x_{t+\Delta t}^{FM}$ is generated by a perfectly pre-trained FM model. As a result, Consistency-FM can learn from the underlying probability path without accessing the ground truth velocity. This demonstrates that the assumptions made in Theorem 1 do not hinder the practical applicability of our method.

Q4: The role of the second term in Equation 6 needs further explanation.

A4: It should be noted that our velocity consistency involves two terms in our loss function for differnt reasons. The first term is performed in the sample space with our defined straight flows (for improving both performance and effciency), and the second term directly constrains on velocity consistency (for better trade-off between performance and effciency). As shown in the table below, we conduct additional ablation experiments on the CIFAR-10 dataset.

From the results, we can observe that when $\alpha=0$ (using only $\left|f_\theta - f_{\theta^{-}}\right|$, Condition 2), both FID and efficiency can be improved, better than traditional sample-space consistency. With an increase in the weight $\alpha$ for velocity difference $(\left\|v_\theta(t, x_t) - u(t, x_t)\right\|_2^2$, Condition 1), we observed a slight degradation in FID but with the significant enhanced training efficiency. This demonstrates that the sample space defined with our velocity consistency (first term) indeed improves both sampling quality and training efficiency while the direct constraint on velocity space (second term) can result in a better trade-off between model performance and convergence. Therefore, both loss are critical for our Consistecy-FM.

Thank you for your detailed feedback and for taking the time to review our paper. We appreciate your valuable insights and suggestions. Please see below for our responses to your comments and we have updated the manuscript to reflect these improvements and clarifications.

Q1: Visual quality is judged by humans based on a few samples, and diversity is also a critical metric.

A1: In addition to our visualized comparisons, we also provide quantitative results on text-to-image generation in the Table 5 and Table 6 of our paper, and we can find that our Consistency-FM significantly outperforms previous methods, demonstrating our superiority. To follow your suggestions, we here measure the generation diversity as [1] for more comprehensive evaluations. Specifically, given a set of input conditions, we first compute the pairwise cosine similarity matrix K among generated images with the same condition, using SSCD [2] as the pretrained feature extractor. The results are then aggregated for different conditions using two methods: the Mean Similarity Score (MSS), which is a simple average over the similarity matrix K , and the Vendi Score [3], which is based on the Von Neumann entropy of K. We list the results in the table below. We also update this results in the Table 7 of our revised manuscript, thanks for your suggestions.

From the results, we can find that our Consistency-FM is consistently better than previous methods regarding the diversity metric, demonstrating the expressiveness of our velocity consistency and multi-segment design.

[1] Sadat, Seyedmorteza, et al. "CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling." ICLR 2024.

[2] Pizzi E, Roy S D, Ravindra S N, et al." A self-supervised descriptor for image copy detection." CVPR 2022.

[3] Friedman D, Dieng A B. "The Vendi Score: A Diversity Evaluation Metric for Machine Learning." TMLR 2022.

Q2: Comparisons of training time to model convergence are important for a comprehensive evaluation but are missing in this paper. Furthermore, I suggest the authors provide a unified metric to measure both quality and speed.

A2: We appreciate your insightful suggestion. Below, we provide a comparison of training times to model convergence on the CIFAR-10 dataset for Consistency-FM, Consistency Model, and RectifiedFlow:

From the comparison, we can find that our Consistency-FM is indeed more efficient for training due to our proposed velocity consistency.

Regarding the unified metric to measure both quality and speed, we acknowledge the importance of such a metric for comprehensive evaluation. Currently, there is no widely accepted unified metric, and to ensure fair comparison, we have not used one in our study. However, we recognize the value of this metric and will consider designing a unified metric to measure both quality and speed in future work. Thank you for this valuable suggestion.

Thank you for your detailed feedback and for taking the time to review our paper. We appreciate your valuable insights and suggestions. Please see below for our responses to your comments and we have updated the manuscript to reflect these improvements and clarifications.

Q1: Why is there a need to use Condition 1 in the loss? Some ablation studies on this could be helpful.

A1: It should be noted that our velocity consistency involves two terms in our loss function for different reasons. The first term is performed in the sample space with our defined straight flows (for improving both performance and efficiency), and the second term directly constrains on velocity consistency (for better trade-off between performance and efficiency). As shown in the table below, we conduct additional ablation experiments on the CIFAR-10 dataset.

From the results, we can observe that when $\alpha=0$ (using only $\left|f_\theta - f_{\theta^{-}}\right|$, Condition 2), both FID and efficiency can be improved, better than traditional sample-space consistency. With an increase in the weight $\alpha$ for velocity difference ($\left\|v_\theta(t, x_t) - u(t, x_t)\right\|_2^2$, Condition 1), we observed a slight degradation in FID but with the significant enhanced training efficiency. This demonstrates that the sample space defined with our velocity consistency (first term) indeed improves both sampling quality and training efficiency while the direct constraint on velocity space (second term) can result in a better trade-off between model performance and convergence. Therefore, both loss are critical for our Consistency-FM.

Q2: In Lemma 1 both conditions are imposed with arbitrary t and s. Why in the loss only neighboring points are considered? How does the performance depend on the choice of Δt?

A2: When each point and its neighboring point satisfy the conditions in Lemma 1, the conditions are also satisfied for arbitrary points at t and s. Arbitrarily choosing t and s implies that during training, Δt = s - t will be arbitrary. However, our preliminary experiments indicated that training converged more unstably when Δt was either excessively large or too small. Since we multiply the timestep by 999 before converting it into a positional time embedding, following the implementation of Rectified Flow [1], and based on our early experimental results, we use Δt = 1e-3 in the final experiments. Below is a table showing how performance depends on the choice of Δt:

[1] Liu, Xingchao, and Chengyue Gong. "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow." International Conference on Learning Representations, 2023.

Q3: It seems that Theorem 1 cannot provide intuition on why training from scratch works. Could the authors clarify this?

A3: Thank you for your insightful comment. We provide an intuition on why training Consistency-FM from scratch works in the updated Appendix A.6, highlighted in blue. By proving Theorem 3, we show that training Consistency-FM from scratch using the conditional trajectory $x_t, x_{t+\Delta t}$ is equivalent to training with a perfectly pre-trained FM trajectory, as $\Delta t \approx 0$. The equation below is from the updated Theorem 3. Here, $x_{t+\Delta t}^{FM}$ is generated by a perfectly pre-trained FM model.

$$E|| f\_\theta(t,x\_{t}) - f\_{\theta^-}(t+\Delta t,x\_{t+\Delta t}^{FM})||\_2^2 = E|| f\_\theta(t,x\_{t}) - f\_{\theta^-}(t+\Delta t,x\_{t+\Delta t})||\_2^2 + o(\Delta t)$$

As a result, Consistency-FM can learn from the underlying probability path without accessing the ground truth velocity.

Q4: Based on Theorem 1, there is a trade-off between consistency and matching the ground truth vector field. However, this trade-off is not observed in the experiments, where Consistency-FM is always strictly better than the prior work. Could the authors comment on this?

A4: As demonstrated in the table provided in above Answer 1, Consistency-FM achieves a balance between performance and training efficiency by navigating the trade-off between consistency and matching the ground truth vector field. This equilibrium enables Consistency-FM to effectively optimize both model performance and training efficiency.

Q5: Some ablation studies for the multi-segment Consistency-FM, e.g. on the number of segments and their sizes, would definitely improve the presentation quality.

A5: Thank you for the suggestion. As shown in the table below, sample quality improves with an increase in the number of segments for the multi-segment Consistency-FM model on the CIFAR-10 dataset. Here the number of evaluations (NFE) is equal to the number of segments for the multi-segment Consistency-FM.

This demonstrates that increasing the number of segments enhances the quality of generated samples, validating the effectiveness of the multi-segment approach in improving model performance.

Q6: From the paper and the implementation details in section 4 it is not clear, which configuration of the method was used for the experiments. E.g. what is the value of $\Delta t$​ ? Was multi-segment consistency-fm used? Is the number of the segments equal to the NFE in Tables?

A6: We apologize for the oversight in the implementation details. We set $\Delta t = 0.001$ in all experiments. Multi-segment Consistency-FM is applied only for experiments where the number of function evaluations (NFE) is greater than 1. The NFE is equivalent to the number of segments for the multi-segment Consistency-FM. We have added these details to the manuscript for clarity, highlighted in blue. Thank you for bringing this to our attention.

Q7: Assuming that the models are trained with the multi-segment approach, can a model trained with $K$​​ segments be used with a different number of NFE at inference?

A7: Yes, a model trained with $K$ segments can alternate denoising and noise injection steps similar to Consistency Models in some segments and achieve a different number of evaluations (NFE) at inference. For example, in the i-th segment, we use the following formula to denoise:

$$x_{i / K}=x_{(i-1) / K}+ \frac{1}{K} * v_\theta^i\left(\frac{(i-1)}{K}, x_{(i-1) / K}\right)$$

Then we inject noise and denoise again using the formula, but replacing $x_{(i-1) / K}$ with $x'\_{(i-1) / K}$.

Q8: How exactly is the GAN loss integrated in the training?

A8: We integrate the GAN loss in training our Consistency-FM similarly to CTM [1], which is not our main contribution and thus we did not discuss it in our paper. More specifically, we additionally train a discriminator to distinguish between our generated (fake) image and real image, and our generator (sampler) tries to cheat the discriminator by learning the real data distribution as much as possible. This adversarial loss shows that a combination of reconstruction and adversarial losses is beneficial for generation quality.

[1] Kim, Dongjun, et al. "Consistency trajectory models: Learning probability flow ode trajectory of diffusion." arXiv preprint arXiv:2310.02279 (2023).

Q9: Some points in the proofs could be expanded. E.g. step 3 in Equation 19.

A9: We have extended the proof of Theorem 1 in the updated Appendix A.3. The revised sections are highlighted in blue:

$$\begin{aligned} f_\theta(t,x_t) - f_\theta(t+\Delta t , x_{t+\Delta t}) &= x_t +(1-t)*v_\theta(t,x_t) - (x_{t+\Delta t} + (1-t-\Delta t)*v_\theta(t+\Delta t, x_{t+\Delta t}) ) \\\\ &= x_t - x_{t+\Delta t} + (1-t) * v_\theta(t,x_t) - (1-t-\Delta t) * v_\theta (t+\Delta t , x_{t+\Delta t}) \\\\ &= -\int_{t}^{t+\Delta t} u(s,x_s)ds + \Delta t * v_\theta (t+\Delta t, x_{t+\Delta t}) + (1-t) * (v_\theta (t,x_t)-v_\theta (t+\Delta t, x_{t+\Delta t}))\\\\ &= \Delta t * (v_\theta (t+\Delta t, x_{t+\Delta t}) - u(t^{'},x_{t^{'}})) + (1-t) * (v_\theta (t,x_t)-v_\theta (t+\Delta t, x_{t+\Delta t})) \\\\ &=\Delta t * (v_\theta (t+\Delta t, x_{t+\Delta t}) - u(t^{'},x_{t^{'}})) - (1-t) * \Delta t * (\partial_t v_\theta (t,x_t)-u(t,x_t)\partial_x v_\theta (t, x_{t})) + O((\Delta t)^{2}) \\\\ &=\Delta t * (v_\theta (t+\Delta t, x_{t+\Delta t}) - u(t,x_{t})+O(\Delta t)) - (1-t) * \Delta t * (\partial_t v_\theta (t,x_t)-u(t,x_t)\partial_x v_\theta (t, x_{t})) + O((\Delta t)^{2}) \\\\ &= \Delta t * (v_\theta (t+\Delta t, x_{t+\Delta t}) - u(t,x_{t}) - (1-t) * (\partial_t v_\theta (t,x_t)-u(t,x_t)\partial_x v_\theta (t, x_{t})) )+ O((\Delta t)^{2}) \end{aligned}$$

Thank you for pointing out the typos in our paper. We have corrected them accordingly. We appreciate your thorough review and valuable feedback.

Q3: Illustrating how velocity-space consistency improves model performance compared to traditional sample-space consistency and clarifying the impact of velocity-based consistency on the model’s stability.

A3: It should be noted that our velocity consistency involves both terms of our loss function, as discussed in our response to Q1. The first term is performed in the sample space with our defined straight flows (for improving both performance and effciency), and the second term directly constrains on velocity consistency (for better trade-off between performance and effciency). As shown in the table below, we conduct additional ablation experiments on the CIFAR-10 dataset.

From the results, we can observe that when $\alpha=0$ (using only $\left|f_\theta - f_{\theta^{-}}\right|$), both FID and enficiency can be improved, better than traditional sample-space consistency. With an increase in the weight $\alpha$ for velocity difference ($\left\|v_\theta(t, x_t) - u(t, x_t)\right\|_2^2$), we observed a slight degradation in FID but with the significant enhanced training efficiency. This demonstrates that the sample space defined with our velocity consistency (first term) indeed improves both sampling quality and training efficiency while the direct constraint on velocity space (second term) can result in a better trade-off between model performance and convergence.

**Q4: Provide CIFAR-10 experiments with a distillation sampling variant, the ImageNet 64x64 experiments with a direct consistency training variant. Both the CIFAR-10 and ImageNet experiments omit several key baselines $2$

$3$

$4$ .**

A4: Thanks for your suggestions. We provide additional performance comparisons with distillation sampling variant on CIFAR-10 (Table 1) and with direct consistency training variant on ImageNet 64 × 64 (Table 2). We have now included the key baselines [2], [3], [4] in Table 3 and Table 4. We also update these results in Tab.1 and Tab.2 of our manuscript which are highlighted in blue for clarity.

Table 1: Performance comparisons with distillation sampling variant on CIFAR-10.

Table 2: Performance comparisons with direct consistency training variant on ImageNet 64 × 64.

Table 3: Performance comparisons on CIFAR-10.

Table 4: Performance comparisons on ImageNet 64 × 64.

[2] Berthelot, D., Autef, A., Lin, J., Yap, D.A., Zhai, S., Hu, S., Zheng, D., Talbott, W. and Gu, E., 2023. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.04248.

[3] Song, Y. and Dhariwal, P., 2023. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189.

[4] Heek, J., Hoogeboom, E. and Salimans, T., 2024. Multistep consistency models. arXiv preprint arXiv:2403.06807.

Q5: Include an analysis of how sample quality changes with different NFE steps, especially for the multi-segment Consistency-FM model.

A5: As shown in the table below, sample quality improves with an increase in the number of evaluations (NFE) for the multi-segment Consistency-FM model on CIFAR-10.

This demonstrates that increasing NFE steps enhances the quality of generated samples, validating the effectiveness of the multi-segment approach in improving model performance.

We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feedback. Please see below for our responses to your comments and we have updated the manuscript to reflect these improvements and clarifications.

Q1: Clarify the difference between Consistency-FM and the standard consistency model regarding the final consistency loss. And what is the reasoning behind the weight α for velocity difference is set to 1e-5, and how does it impact the model’s performance?

A1: It is noted that our Consistency-FM defines straight flows with velocity consistency that innovatively bridges consistency models and flow matching models in a simple yet effective manner:

Novelty and effectiveness of our first term: The term $\left|f_\theta - f_{\theta^{-}}\right|$ is derived from Lemma 1, which for the first time defines straight flows with velocity consistency. In Equation (5), these two conditions are equivalent: Condition 1 explicitly enforces velocity consistency, while Condition 2 leads to $\left|f_\theta - f_{\theta^{-}}\right|$. This form is simpler but more effective than the standard consistency model, which has sufficiently demonstrated in Table 1 and Table 2. Additionally, Theorem 1 proves that $\left|f_\theta - f_{\theta^{-}}\right|$ helps reduce the discrepancy between the learned velocity $v_\theta(t, x_t)$ and the ground truth velocity $u(t, x_t)$. Therefore, the final generation performance can be effectively enhanced under flow matching framework, which is also proved by empirical results in large-scale text-to-image generation (Table 6).

Novelty and effectiveness of our second term: Regarding the second term $E_{t, p_t}\left\|v_\theta(t, x_t) - u(t, x_t)\right\|_2^2$ in our final loss function, it is also new to FM methods and is designed to improve training efficiency without significantly degrading model performance. The weight α=1e-5 is empirically chosen based on early experiments because we find that the first and second terms in the loss function differ by several orders of magnitude, multiplying the second term by 1e-5 can effectively balance the magnitudes of the two terms. To further enhance the model's expressiveness, we propose to using a multi-segment approach. By dividing the probability flow ODE trajectory into multiple segments, each segment can be more accurately approximated using straight flow.

Based on both theoretical and empirical analysis, our Consistency-FM not only improves generation performance but also enhances training efficiency by combining the two novel terms in final loss function.

Q2: Provide insights into whether these improvements are statistically significant, or if they are due to the flow matching model itself? To clarify this, a baseline using a standard consistency loss (sample space only) in combination with the flow matching model is needed.

A2: These are really good questions and suggestions. Switching from ODE trajectories derived from diffusion models to flow matching models is indeed one of the reasons for the performance improvement. This improvement is specifically due to the formula we derived from the straight flow theory, which is particularly well-suited for this approach. We conduct an additional experiment on CIFAR-10 dataset, where we use only sample-space consistency loss in combination with the flow matching model. We finally achieve an FID of 5.29, much better than the FID of 5.83 with consistency model.

Notably, as shown in Table 6 of our paper, Consistency-FM achieves an FID of 11.02 through one-step generation on the MS COCO 2014 dataset. The Phased Consistency Model uses a standard consistency loss for one-step generation. In comparison, the Phased Consistency Model achieves an FID of 17.91 (much worse than our FID of 11.02) in Table 2 of [1] through one-step generation on the same dataset and with the same backbone. Thus our improvement over standard consistency model is more significant in large-scale scenarios, demonstraing the effectiveness and superiority of our new FM method.

[1] Wang, Fu-Yun, et al. "Phased Consistency Model." NeurIPS 2024.

Dear reviewer AJPE:

We sincerely appreciate the time and effort you dedicated to reviewing our paper. In response to your concerns, we have conducted additional experiments and provided an in-depth analysis to demonstrate both effectiveness and efficiency of our proposed Consistency-FM during the discussion period.

As the discussion period concludes in two days, we kindly request, if possible, that you review our rebuttal at your convenience. Should there be any further points requiring clarification or improvement, please know that we are fully committed to addressing them promptly. Thank you once again for your invaluable contribution to our research.

Warm regards,

The Authors

Thank you to the authors for their additional experiments and discussions. While these efforts address some of my concerns, my primary issue regarding the effectiveness of the additional velocity-space consistency remains unresolved, which directly affects the perceived novelty of this work. Specifically, the provided sample-space-only experiment achieves an FID score of 5.29, outperforming all configurations that utilize velocity-space consistency with a positive $\alpha$. The argument for training efficiency is also unpersuasive, as one could similarly argue that fewer training iterations with sample-space-only consistency loss could yield comparable FID scores.

Regarding the novelty of the sample-space consistency loss itself for flow-matching models, its contribution appears limited. Rectified flow is a specific case within the broader class of diffusion models, with the forward process sharing a formulation akin to $x_t = a_t x_0 + b_t \epsilon$ [2]. Additionally, velocity prediction in flow models and noise prediction in diffusion models are interchangeable [2, Eq. 12]. As a result, the proposed sample-space loss function closely resembles the original consistency function introduced in Consistency Models [1, Eq. 7 and Eq. 9]. Substituting $f_\theta$ from the consistency model for diffusion models with flow-matching models, as described in [2, Eq. 13 and Eq. 14], is a straightforward extension. Thus, the novelty of this paper seems limited, and I am inclined to maintain my current score.

[1] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023, July). Consistency Models. International Conference on Machine Learning, 32211–32252. PMLR.

[2] Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., & Podell, D. (2024, March). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. Forty-First International Conference on Machine Learning.