Implicit Dynamical Flow Fusion (IDFF) for Generative Modeling

Thank you for your detailed review and valuable feedback. We appreciate your recognition of IDFF’s novelty and effectiveness and your suggestions for enhancing clarity and comprehensiveness.

Some parts of the paper need improvement in their descriptions. For instance...

The introduction of the SDE (detailed in Appendix A.1.1) primarily supports the proof of continuity. The solutions $\boldsymbol{x}_t$ of this SDE are governed by the Fokker-Planck equation, which is instrumental in the proof process. Importantly, as shown in [2], the chosen SDE shares the same marginal distributions as the probability ODE paths, ensuring consistency in the marginal distribution. Your observation is correct—the SDE plays a key role in demonstrating that the introduction of the momentum term does not ultimately affect the marginal probability density path $p_t(\boldsymbol{x}_t)$.

Could you provide a clearer explanation of the transition..

While $\tilde{\boldsymbol{v}}_t$ provides flexibility by incorporating momentum, it does not inherently satisfy the probability flow constraint, which requires alignment with the continuity equation. To ensure this constraint is met, the IDFF framework refines $\tilde{\boldsymbol{v}}_t$ into a new vector field $\boldsymbol{w}_t$. This refinement allows IDFF to retain the flexibility provided by leveraging momentum while ensuring that the theoretical constraints required for flow matching (continuity constraint) remain valid and consistent throughout. We will enhance Section 3 to provide clearer explanations and context for the transition from $\tilde{\boldsymbol{v}}_t$ to $\boldsymbol{w}_t$.

[2] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.

Some experiments in the paper are insufficient. For example, in the image generation task...

We appreciate your suggestion to benchmark IDFF against Rectified Flow and will include a comparison and the proper citation in our analysis of the image generation task. Moreover, we recognize that [1] shares a robust conceptual connection with Independent Coupling Flow Matching (I-CFM). Specifically, when the conditional probability path p_t​ is a Dirac delta function (i.e., σ=0), I-CFM becomes equivalent to Rectified Flow. In this case, the reported FID score is 13.68 for Rectified Flow with NFE=10 [1]. We have also computed the FID score for simple FM, equal to 14.36, with NFE=10. These results will be added to the revised version of the paper to ensure a more thorough comparison.

These revisions will enhance the paper’s clarity and strengthen the experimental results, addressing your concerns comprehensively.

Thank you again for your constructive feedback; please let us know if we have appropriately answered your questions and if you have any follow-ups on the same.

Thanks for the author's response, but I believe it does not address my concerns.

Regarding my first question, the key statement in the author's response is: “Furthermore, in Lemma 1, we assume $v_t$ is the vector field that generates the probability path $p_t$.” This directly assumes that $v_t$ and $\tilde{v}_t$ have the same marginal probability distribution, which is a strong assumption. The authors should provide further clarification to support this point. From the phrase “To expand on the above explanation further,” the author's response primarily elaborates on the properties of the continuity equation itself, namely, “the flow of probability neither creates nor destroys mass as it moves.” However, I do not believe this adequately addresses my concern about how $v_t$ and $\tilde{v}_t$ can produce the same marginal probability density $p_t$.

For my second question, I suggest the authors calculate the precision and recall to quantitatively demonstrate that IDFF does not suffer from mode collapse. I do not believe the generated images in Figures 7–12 are sufficient to prove the absence of mode collapse in IDFF.

Additionally, similar to Reviewer U2S3, I also have questions about how IDFF leverages HMC to reduce NFE. The statement in lines 215–217 of the paper, “These properties lead to faster convergence to the target distribution with fewer function evaluations,” lacks direct evidence. The properties of HMC, such as “reducing random walk behavior and overcoming local energy barriers,” are not clear to me in terms of how they directly reduce the number of steps required to solve the ODE numerically.

This work is not convincing enough, both in terms of motivation and theoretical proof. I believe the authors need to present their motivation more clearly and provide a more rigorous proof for their theory.

However, I do not believe this adequately addresses my concern about how vt and v~t can produce the same marginal probability density pt.

Thank you for highlighting this concern. For $v_t(x)$ to produces the marginal density $p_t(x)$ it must satisfy:

$

v_t(x) = \mathbb{E}_{q(z)} \left[\frac{v_t(x|z) p_t(x|z)}{p_t(x)}\right]. \tag{**}

$

To show that $\tilde{v}_t(x)$ satisfies the same marginal probability we can write $\tilde{v}$ as a function of $v$, substitute it into the above, and obtain a condition on $\xi$. We can then show that $\xi=\gamma\log p_t(x)$ satisfies the necessary condition. The full proof is provided in the following comment. We will include the proof in Appendix A.1.1 of the manuscript to explicitly showcase that the transformed vector field $\tilde{v}_t(x)$, defined by Equation (6), produces the same marginal probability density $p_t(x)$ as the original vector field $v_t(x)$.

We hope this clarifies the concern about the theoretical rationale behind our work. We will work to incorporate this argument better in the main paper as well.

For my second question, I suggest the authors calculate the precision and recall to quantitatively demonstrate that IDFF does not suffer from mode collapse.

We appreciate the reviewer's suggestion to calculate precision and recall as quantitative measures to assess whether IDFF suffers from mode collapse. In response, we evaluated IDFF on CIFAR-10 with NFE=10, resulting in a precision of 0.76 and a recall of 0.54. These results indicate that IDFF effectively avoids mode collapse, as a high recall demonstrates the model's ability to generate diverse samples that cover the true data distribution.

For comparison, recall values from other diffusion-based models under similar evaluation settings are as follows (adapted from [4]):

DDIM with NFE=50: Recall = 0.53

DDPM with NFE=1000: Recall = 0.57

Notably, IDFF achieves a higher recall than DDIM despite requiring significantly fewer function evaluations (NFE=10 compared to NFE=50). This highlights IDFF's robustness in maintaining diversity across generated samples.

[4] Zheng, Huangjie, et al. "Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders." arXiv preprint arXiv:2202.09671 (2022).

We appreciate your recognition of IDFF’s contributions and the constructive recommendations to improve clarity, particularly regarding the momentum term’s role in reducing NFEs. Below, we address your concerns.

Please include Hamiltonian Monte Carlo in the background section (Section 2)...

We appreciate the insightful feedback and agree that the explanation regarding the momentum term's role in reducing NFEs needs to be clarified. We will address this comprehensively in the revised paper. In section 2, we will add background on HMC and motivate why it provides new ideas for flow-matching models. In 3.1, we will highlight the conceptual connections between IDFF and HMC.

To elucidate further:

Hamiltonian Monte Carlo (HMC) is a powerful sampling algorithm designed to improve efficiency in exploring complex probability distributions by introducing auxiliary momentum variables. In HMC, particle dynamics are simulated in a potential field using Hamiltonian dynamics, which describe the total energy of a system through the Hamiltonian function:

where $U(\mathbf{x})$ represents the potential energy related to the target distribution, and $K(\mathbf{p}) = \frac{1}{2} \mathbf{p}^T \mathbf{M}^{-1} \mathbf{p}$is the kinetic energy, with $\mathbf{M}$ being the mass matrix. The dynamics of this system evolve according to:

These dynamics preserve the system's total energy, enabling efficient exploration of the probability space. HMC achieves this by leveraging momentum to facilitate directed movement, reducing random walk behavior, and overcoming local energy barriers. These properties lead to faster convergence to the target distribution with fewer evaluations.

HMC generates a sample from some target distributions by following Hamiltonian dynamics in an extended state space $(\mathbf{x}, \mathbf{p})$, where $\mathbf{x}$ represents the position and $\mathbf{p}$ represents momentum. IDFF attempts to design a flow-matching procedure with a similar intuition in flow-matching models.

The momentum term in IDFF enables it to approximate the target distribution effectively with fewer sampling steps, reducing the number of function evaluations (NFEs) required while maintaining high sample fidelity.

This connection highlights how IDFF inherits HMC’s momentum-based exploration benefits while maintaining the computational advantages of Conditional Flow Matching (CFM). By leveraging $\boldsymbol{\xi}_t$, IDFF accelerates convergence without compromising the quality of generated samples, directly addressing the need for efficient sampling with reduced NFEs.

We will ensure this discussion is included in the revised manuscript to strengthen the justification for the method. Thank you again for your constructive feedback; please let us know if we have appropriately answered your questions and if you have any follow ups on the same.

The proposed loss ...

There are several critical differences with [2]:

1-The motivations behind the two approaches differ significantly. In [2], the focus is on exploring the connection between the Schrödinger Bridge (SB) problem and entropic OT, framing the SB as a mixture of Brownian bridges. Our work's derivation draws inspiration from the dynamic processes of HMC sampling to improve sampling speed while preserving or enhancing sample quality.

2- Practically, [2] training two separate models: one dedicated to learning the scoring term, $\boldsymbol{s}(t, \boldsymbol{x}_t)$, and another for the drift term, $\boldsymbol{v}(t, \boldsymbol{x}_t)$. These two components are combined during the sampling phase. This necessitates the need for separate training and integration steps. In contrast, IDFF employs a single unified model to train for both the score and flow terms simultaneously. This improves computational efficiency.

3- [2] defines the drift term in an SDE as the sum of a vector field and a scoring term, expressed as:

Where $\boldsymbol{v}(t, \boldsymbol{x}_t)$ represents a learned vector field, and $\boldsymbol{s}(t, \boldsymbol{x}_t)$ is a learned score function. In contrast, we propose a novel vector field inspired by HMC sampling dynamics, formulated as:

IDFF formulation introduces a denoising model in the sample space, $\hat{\boldsymbol{x}}_1$, and a momentum variable, $\hat{\epsilon}$, for the vector field generation.

4- While [2] learns via a loss function in the vector space, our approach applies the loss directly in the input space. To study the impact of this modeling choice, an ablation study comparing the FID scores on the CIFAR-10 dataset at NFE=10. When the learning objective was applied in the vector space, the FID score was 7.32, while applying it in the input space reduced the score to 5.87. Additionally, we compared these results with the FID score of the SB method, which was 10.13 for NFE=10. IDFF outperformed the SB approach, highlighting that the design choices in IDFF contribute empirically to enhancing sample quality.

We will summarise the discussion mentioned above in the updated manuscript.

In the CIFAR10...

We have compared IDFF with DPM++, DPM-v3-solver, and UniPC under the NFE=5 setting to address this. The results show that IDFF outperforms these methods in the CIFAR10 dataset.

of less importance ...

As demonstrated by [7], in CFMs, the target vector fields don't need to be explicitly conditioned on $\boldsymbol{x}_1$. Instead, the network can depend on $\boldsymbol{x}_1$. This results in a loss function similar to the original FM loss (equations (5) in [7]), differing only by a constant term, which does not affect the optimization process or gradient calculations. We will revise the manuscript to clarify this explanation and address potential ambiguities in interpreting Equation (5).

Regarding your second point, the flow matching loss in Equation (5) becomes nearly intractable when not conditioned on the initial or target distributions. While FM has been successfully applied to large-scale tasks like image and video generation, these applications typically involve CFM or variations, which incorporate conditioning to address this intractability. [1] have used conditioning and coupling methods, such as OT-CFM, to make the problem more tractable for their respective tasks. We will clarify this distinction in the revised manuscript.

In the paragraph ...

Thank you for pointing this out. In IDFF, as detailed in Section 3.2, we utilised OT for coupling. In Equation (5) of the Background section, we outlined the general form of the CFM loss, which can accommodate various coupling terms, not just independent ones, as demonstrated in [1]. We will clarify this explanation in the manuscript. Thank you again for your constructive feedback; please let us know if we have appropriately answered your questions and if you have any follow-ups.

[7] Lipman, Yaron, et al. "Flow matching for generative modeling." arXiv preprint arXiv:2210.02747 (2022).

First, I want to note this table kind of avoids my comment, since in the paper a 10 NFE was compared and hence I was referring to the performance at 10 NFE. Second, can the author explain what are the models being compared in this table? That is on which CIFAR10 model does the other solvers were tested on?

We apologize for the misinterpretation of your earlier comment. IDFF primarily focuses on reducing NFE (Number of Function Evaluations) and demonstrating performance at lower NFE values (e.g., NFE=5), which is why we initially reported those results. Based on your suggestion, we have included performance results for NFE=10 in the updated table below. Additionally, we have included wall-clock times to provide a clearer comparison across NFEs:

The reported FID values follow the experimental setup outlined in [5], and evaluations were conducted using similar backbones to maintain consistency and comparability.

All models were evaluated on the CIFAR-10 dataset using the same architecture as described in our experimental setup. Specifically, for IDFF, we employed a CNN-based UNet architecture to model $\hat{\mathbf{x}}_1(\mathbf{x}_t, t; \theta)$ and $\hat{\epsilon}(\mathbf{x}_t, t; \theta)$. To ensure fairness, all solvers, including DDPM, DDIM, and FastDPM, were tested using this shared architecture. This ensures equitable assessments of both sample quality and computational efficiency. At NFE=5, IDFF achieves a significantly better FID score (10.97) compared to UniPC (23.52) and DPM-Solver-v3 (12.21), while also boasting the fastest wall-clock time (0.34 seconds) among all solvers. This highlights IDFF's ability to generate high-quality samples with minimal computational overhead, making it ideal for real-time applications. Even at NFE=10, where UniPC slightly edges out in FID (2.85) alongside DPM-Solver-v3 (2.91), IDFF remains competitive with a reasonable FID (5.87) and the fastest wall-clock time (0.52 seconds), demonstrating its efficiency and scalability. These results suggest that IDFF hits a balance between sample quality and computational speed that lends itself to speed.

Finally, what does the author mean by " these models because they require significantly higher memory and computational resources"? Are the models in the table above are not of comparable number of parameters to the ones in the paper?

The higher-order solvers mentioned in the table, such as those described in Algorithms 1 and 2 of UniPC [6], require additional memory and computational resources due to their design.

Specifically: Higher-order solvers cache multiple intermediate values of $\epsilon_t$ to perform higher-order approximations. This caching process increases both memory requirements and computational overhead during sampling. In contrast, our proposed model generates samples without caching any intermediate calculations in memory. Instead, $\hat{\mathbf{x}}_1(\mathbf{x}_t, t; \theta)$ and $\hat{\epsilon}(\mathbf{x}_t, t; \theta)$ are computed simultaneously during the sampling process. This "on-the-fly" approach eliminates the need for storing intermediate values, resulting in significantly reduced memory and computational requirements compared to higher-order solvers. We hope this clarifies the differences and rationale behind the computational efficiency of IDFF.

[6] Zhao, Wenliang, et al. "Unipc: A unified predictor-corrector framework for fast sampling of diffusion models." Advances in Neural Information Processing Systems 36 (2024).

We are still working on a response to enhance the clarity of the method's exposition and to address your last comment with greater precision.

Figure 2 C) is a bit confusing by itself. Only after reading Algorithm 2 ...

We acknowledge that Figure 2(C) could be more self-explanatory. To address this, we will revise the figure to include clearer annotations that better illustrate the sequence of steps within the IDFF sampling process, making it more intuitive and self-contained.

The authors compare image generation quality w.r.t sampling strategies like DDIM and DPM and show that their model achieves the best quality...

We thank the reviewer for suggesting comparisons with EDM and Rectified Flow. To address this, we have added a comparison of FID scores with NFE=5 for EDM in the newly added results table for the CIFAR-10 dataset.

Table: Additional FID Performance Comparison (NFE=5)

Our results demonstrate that IDFF achieves superior performance under this setup. Additionally, we evaluated FID scores for EDM across various NFE values

NFE=6, FID=103.86

NFE=8, FID=39.66

NFE=10, FID=16.57

NFE=12, FID=7.59

The proposed IDFF model offers notable advantages over both EDM and Rectified Flow. While EDM employs equilibrium-based diffusion processes to generate high-quality samples, IDFF introduces a momentum term, $\boldsymbol{\xi}_t$, to guide sample generation through an enhanced vector field. This approach accelerates the sampling process while preserving sample quality by leveraging dynamics akin to Hamiltonian Monte Carlo, focusing on efficiency in the sampling process.

Rectified Flow adjusts flow dynamics to accelerate convergence. IDFF differs by modeling the flow field in input space while learning momentum variables through a score-matching loss. As the synthetic data demonstrate, the momentum variables pre-empt how the data's dynamics should evolve, thereby reducing the NFEs required for high-quality image generation.

While EDM is a powerful approach, IDFF's momentum-guided dynamics and time-weighted sampling strategy render it a complementary and efficient alternative for rapid sampling and generative processes. We find that IDFF is particularly effective in time-series generation, a setting not often considered by other models. As confirmed by our experimental results, it demonstrates significant improvements across various temporal scales. We hope this extended analysis provides a clearer understanding of IDFF's strengths relative to these state-of-the-art approaches.

Are there any limitations to the author's methods ...

While IDFF's introduction of the momentum term increases training complexity compared to CFMs in practice, this is effectively balanced by a substantial reduction in the number of sampling steps required during inference. We will expand the discussion in the paper to present a balanced perspective on these trade-offs and their implications for performance. Notably, IDFF leverages the strengths of its underlying CFM models to produce high-quality samples (FID < 10) with just 500K iterations of training —a significant improvement over the millions of iterations required by diffusion-based models and comparable to CFMs. We will include a figure to visualize this improvement for clarity. Thank you again for your constructive feedback; please let us know if we have appropriately answered your questions and if you have any follow-ups.

Thank you authors for sincere rebuttal. The authors have provided additional explanation for most of my concerns, with additional results I still stand with my current rating.