Learning to Integrate Diffusion ODEs by Averaging the Derivatives

We sincerely thank the reviewer for the valuable feedback. We are grateful for the recognition of our paper's clear organization and the interesting connections provided by our theory. We address the reviewer's concerns below.

Regarding Weakness 1

Reviewer: The comparison to and discussion of previous consistency models on ImageNet-256$\times$256 appears to be insufficient. The paper only mentions training instability issues encountered by previous consistency models on ImageNet-256$\times$256. It remains unclear whether appropriate hyperparameters and settings were used for training these models. A more detailed discussion of the failure cases of consistency models would enhance the clarity and quality of the paper and highlight the stability of the proposed method.

We thank the reviewer for indicating this important aspect. For more detailed discussion, we focus on the analysis of the target stability. Target stability is a rather crucial factor to train few-step diffusion models, which means the stability of the prediction target in the loss function [1]. The difference in target stability between CM and ours is essential in principle, which cannot be influenced by specific hyperparameters or settings. We apologize for not elaborating on this in the main text. For simplicity, let's compare the targets under the OT-FM interpolant ($d$ denotes some distance metric):

• Diffusion Models (Flow Matching): $\mathcal{L}(\theta)=d(\\boldsymbol{v}\_\theta(\\boldsymbol{x}\_t,t)-(-\\boldsymbol{x}_0+\\boldsymbol{z}))$. The target is the velocity estimation $-\\boldsymbol{x}\_0+\\boldsymbol{z}$.
• MeanFlow / CM / CTM (similar): $\mathcal{L}(\theta)=d(\\boldsymbol{f}\_\theta(\\boldsymbol{x}\_t,t,s)-((-\\boldsymbol{x}_0+\\boldsymbol{z})+(s-t)\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s))$. The target is $-\\boldsymbol{x}_0+\\boldsymbol{z} + (s - t)\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)$. The latter item $\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)$ (or the discrete version $\\frac{\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)-\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_{t-\Delta t},t-\Delta t,s)}{\Delta t}$) is highly unstable, which has been substantially studied in [1].
• Our Method: The targets are either the velocity estimation $(-\\boldsymbol{x}\_0+\\boldsymbol{z})$ or the velocity $\\boldsymbol{v}(\\boldsymbol{x}_t,t)$ itself.

Our method is the only one among these to feature a target with stability comparable to that of standard diffusion models. This inherent stability of the prediction target helps explain the robust training behavior we observe. To see the practical target stability, we also conduct an experiment with EDM on CIFAR-10 (the first table below) and also SiT on ImageNet-256 (the second table below), where we track the specific items in the loss functions, and record the standard deviation in the following table.

The statistics shown in the tables clearly support the stability of our method. While, for CMs, the target is unstable, especially when the data has large variance (e.g., the latent space of ImageNet-256). The instability largely affects the training dynamics, potentially leading to worse performance, or even training divergence.

Regarding Weakness 2

Reviewer: The condition for $|s−t|$ in Theorems 2 and 3 appears to be too restrictive for practical few-step diffusions. The magnitude should be $O(L^{−1})$ for Theorem 2 and $O(L^{−1/2})$ for Theorem 3, where $L$ denotes the Lipschitz constant of $v(x_t,t)$ and $f_\theta(x_t,t,s)$, respectively. It is generally not guaranteed that the neural network or the flow is Lipschitz in practical cases. This could be a limitation of the proposed method.

We agree with the reviewer that Lipschitz continuity is not generally guaranteed for a standard neural network in practice. However, we work in the context of diffusion models, where this condition is a necessary condition to apply foundational results like the Picard-Lindelöf theorem, which guarantees the existence and uniqueness of the ODE solution. This provides a rigorous framework for analyzing our algorithm's convergence, stability, and correctness, a practice common to many works in this field. For example, the seminal work, and perhaps the most related to our work, consistency models [2], and consistency trajectory models [3], both include the Lipchitz condition in the Theorems. And the theoretical guarantee is under the worst case; in practice, the strong empirical stability of our method suggests that our training procedure implicitly guides the network to a well-behaved state. The learned function appears to be locally smooth in the relevant regions of the state space, thus closely approximating the conditions required by the theory.

And we also agree with the reviewer that, the requirement on $|s-t|$ affects the performance in extremely low-NFE settings. In this situation, the small numerical error from one step could be amplified in subsequent steps. Here we would like to highlight that it is a key trade-off compared to consistency models in the theoretical design. CMs emphasize global accuracy, but introduce either discretization error or explicit differentiation part in the loss function, which significantly affects the training stability. Our method achieves local accuracy without explicit differentiation, and has a very stable target as diffusion models. We believe this presents a valuable and complementary approach to CMs. And we position our method between fast samplers and diffusion distillation/training methods, as a good trade-off between performance and efficiency.

Regarding Weakness 3

Reviewer: There is subtle missing information and errors in the formulas.

• The definition of the prime symbol ', around Eq. (1) and Proposition 1, is unclear, while I presume it denotes the derivative with respect to time $t$.
• The variables $a$ and $b$ are not used in the statement of Theorem 3 in the main part of the paper. Please clarify the relationship between $[a,b]$ and $[t,s]$ in Theorem 3.
• In the proof of Th.3, there are confusing duplications of the variables, $dr$, in the nested integrations. Please use different variables for the nested integrals, e.g., $dr$ and $d\tilde{r}$.
• $=0$ in the middle of the equation around L554 is possibly a typo.

i) You are correct; the prime symbol denotes the derivative with respect to time $t$. We will explicitly state this definition in the revised paper.

ii) In Theorem 2, we fix a $t$ and move $s$ in a small neighborhood. While in Theorem 3, it requires sampling time pairs where $t$ and $s$ can appear in any order (also see Figure 1). Therefore, we use a interval $[a,b]$ to depict that increasing the distance $|s-t|$ is accompanied by expanding the interval where $t$ and $s$ are randomly sampled rather than simply moving $s$, as discussed in Remark 4.

iii) Thank you for catching this typo. We will use different variables for the nested integrals in our revision to avoid confusion.

iv) Thank you for pointing out this error. We will correct it in the revised manuscript.

[1] Simplifying, Stabilizing and Scaling Continuous-time Consistency Models. Cheng Lu and Yang Song. ICLR 2025

[2] Consistency models. Yang Song et al. ICML 2023

[3] Consistency trajectory models: Learning probability flow ode trajectory of diffusion. Dongjun Kim et al. ICLR 2024

Dear Reviewer,

This is a gentle reminder that the Author–Reviewer Discussion phase ends within just three days (by August 6). Please take a moment to read the authors’ rebuttal thoroughly and engage in the discussion. Ideally, every reviewer will respond so the authors know their rebuttal has been seen and considered.

Thank you for your prompt participation!

Best regards,

AC

We sincerely thank the reviewer for the valuable and constructive feedback. We are also grateful for the recognition of our method as innovative and well-motivated. We will address each of the reviewer's concerns below.

Weaknesses 1 (Question 2)

Our theoretical framework grounded in Theorems 2 and 3 is designed to emphasize local accuracy in the absence of explicit differentiation, which guarantees that our method can work well at short intervals, empirically like 4 or 8 steps. As a result, we position our method between faster samplers and few-step diffusion distillation/training methods, as our method provides good trade-off between performance and efficiency. We provide additional experiments to show the stability and efficiency against other methods. Please also see the response to Reviewer v8s1 regrading Weakness 2.

Weaknesses 2

We agree with the reviewer that Lipschitz continuity is not generally guaranteed for a neural network in practice. However, we work in the context of diffusion models, where it is a necessary condition to apply foundational results like the Picard-Lindelöf theorem, which guarantees the existence and uniqueness of the ODE solution. This provides a rigorous framework for analyzing our algorithm's convergence, stability, and correctness, common to many works in this field. For example, the seminal work, consistency models [1] and consistency trajectory models [2], both include the Lipchitz condition in the Theorems. And the theoretical guarantee is under the worst case; in practice, the strong empirical stability of our method suggests that our training procedure implicitly guides the network to a well-behaved state, thus closely approximating the conditions required by the theory.

For violations, i) if the function is not Lipschitz, the convergence is not guaranteed. However, according to the experiments, our method consistently trains models with great stability and performance, suggesting the function appears to be locally smooth in the relevant regions of the state space. ii) The Lipchitz constant is not small enough. In this situation, the small numerical error from one step could be amplified in subsequent steps. Considering this, we position our method between fast samplers and diffusion distillation/training methods, as a good trade-off between performance and efficiency.

Weaknesses 3 (Question 3)

First, we would like to correct an error in Table 7 of the appendix. The 4- and 8-step FIDs for STEI on CIFAR-10 should be 4.04 and 2.59. We discovered that dropout was not correctly enabled in our previous experiments, contrary to the setting of 0.2 specified in Table 8.

In Table 7, the difference happens mainly between estimating inner or end point (training/distillation achieves similar performance), which can be explained by three aspects. i) Input correctness. For inner-point variants, the input to the training network is the clean data $\boldsymbol{x}_t$. While in end-point variants, the input is the estimated $\\hat{\boldsymbol{x}}\_t$. Fitting a fixed, clean input distribution, is more efficient than fitting an input dependent of the model historical output. ii) Error accumulation path, i.e., the path from clean data to the prediction destination. In inner-point variants, the path is $\boldsymbol{x}_t\to \boldsymbol{x}_s$, while in end-point variants it is $\boldsymbol{x}_r\to \boldsymbol{x}_t \to \boldsymbol{x}_s$. Generally, the path in end-point variants is longer than that in inner-point variants. iii) Model capacity. As per Figure 1 and Algorithm 2, the end-point variants additionally require inversion, which costs part of the model capacity.

Weaknesses 4 (Question 4)

We thank the reviewer for pointing out this omission. We include the proofs for STEI and SDEE below. We will add these to the final manuscript. (\boldsymbol is omitted to save the character count.)

Corollary 4. (STEI) Let $f_\theta(x_t,t,s)$ be a neural network, and let $v(x,t)$ be the true velocity field. Assume $v(x,t)$ is L-Lipschitz continuous in its first argument. If $\mathcal{L}\_{STEI}(\theta)$ reaches its minimum, then for any interval where $|s-t| \le h$ and $h < 3/L$, we have $f_{\theta}(x_{t},t,s)=f(x_{t},t,s)$.

Proof. The loss $\mathcal{L}\_{STEI}(\theta)$ is defined as a sum of a secant term and a diffusion term: $\mathcal{L}\_{STEI}(\theta) = \mathbb{E}[||f_\theta(x_t,t,s)-f_{\theta^-}(\hat{x}\_r,r,r)||\_2^2] + \lambda\mathbb{E}[||f_\theta(x_\tau,\tau,\tau)-(\alpha_\tau'x_0+\sigma_\tau'z)||\_2^2],$ where $\hat{x}\_r = x_t+(r-t)f_{\theta^-}(x_t,t,r)$. For $\mathcal{L}\_{STEI}(\theta)$ to reach the minimum, both two terms must reach the minimum. By Proposition 1 of our paper, the diffusion term ahieves the minimum if and only if $f_\theta(x_t, t, t) = \mathbb{E}(\alpha_t'x_0+\sigma_t'z|x_t) = v(x_t,t).$ Subsitituting this result into the secant term, it becomes the one studied in the proof of Theorem 2. The remainder of the proof follows identically.

Corollary 5. (SDEE) Let $f_\theta(x_t,t,s)$ be a neural network that is L-Lipschitz continuous in its first argument. For any fixed interval $[a, b]\subseteq[0,1]$ with $b-a$ sufficiently small, if $\mathcal{L}\_{SDEE}(\theta)$ reaches its minimum, then $f_\theta(x_t,t,s)=f(x_t,t,s)$.

Proof. The SDEE loss is defined as: $\mathcal{L}\_{SDEE}(\theta) = \mathbb{E}\_{x_0,z,t,s,r\sim\mathcal{U}(t,s)}[||f_\theta(x_r+(t-r)f_{\theta^-}(x_r,r,t),t,s)-v(x_r,r)||\_2^2].$ This loss function is identical to the intermediate objective $\mathcal{L}\_{*}(\theta)$ analyzed in the proof of Theorem 3. The proof, therefore, follows that of Theorem 3 directly.

Weakness 5 (Question 5)

Mathematically, uniform sampling and a non-uniform distribution rebalanced with importance sampling are equivalent. However, in the context of deep learning, the choice of sampling distribution corresponds to how we allocate the training effort across different time points. We apologize for not clarifying this motivation in the main text. Inspired by recent findings in the diffusion model literature (e.g., EDM [3], SD3 [4], FasterDiT [5]), where focusing training efforts on specific time intervals was shown to accelerate convergence, we decided to explore if a similar strategy would benefit our model. However, empirical results indicate that allocating training effort uniformly works best for our method.

Weakness 6

As stated in the main text below Eq. (9), “This approach is inspired by the diffusion objective that leads from Eq. (1) to Eq. (2).”, our use of Monte Carlo estimation for sampling data and time points within one batch is analogous to the standard practice in training diffusion models. We follow this manner and choose the same batch size (e.g., 256 for ImageNet, 512 for CIFAR-10) as these baseline models. The experimental results validate that it is sufficient for stable convergence. Regarding the analysis of convergence rates, to our knowledge, it is still an open research problem in the field of deep generative modeling. To directly address the reviewer's point on sample sizes and computational trade-offs, we conduct ablation study on the effect of batch size on performance. We train models with SDEI and EDM on CIFAR-10 with varying batch sizes and report the 4-step FID. The table below suggests that our method is robust to the batch size. We will add these results to the appendix of our paper.

Weakness 7

We thank the reviewer for indicating the presentation issues. We will correct the typos. For Proposition 1, as stated in the appendix after the proof of Proposition 1, “The above is a basic result in literature of diffusion models [66].” We will also add the attribution and references to the main text as well.

Question 1

The additional complexity is only introduced to the training phase, i.e., the extra forward and backward passes as summarized in Table 1. Other minor computations, such as sampling time steps $s$ and $r$, are negligible. Practical running speed and memory cost can be referred to the response to Reviewer v8s1, regarding Weakness 2. Our method has almost no impact on the inference/sampling phase, as the only architectural modification is the injection of an additional time condition, which has a negligible computational cost. Therefore, it does not affect the claimed efficiency gains.

Question 6

Since few-step diffusion distillation/training methods typically do not modify the model architecture, the running time per NFE is the same as diffusion models. We list some results in the table below (the first is on CIFAR-10, and the second is on ImageNet-256), where we use one A100 GPU with a batchsize of 8.

[1] Consistency models. Yang Song et al. ICML 2023

[2] Consistency trajectory models: Learning probability flow ode trajectory of diffusion. Dongjun Kim et al. ICLR 2024

[3] Elucidating the design space of diffusion-based generative models. Tero Karras et al. NeurIPS 2022

[4] Scaling rectified flow transformers for high-resolution image synthesis. Patrick Esser et al. ICML 2024

[5] FasterDiT: Towards faster diffusion transformers training without architecture modification. Jingfeng Yao et al. NeurIPS 2024

Dear Reviewer,

This is a gentle reminder that the Author–Reviewer Discussion phase ends within just three days (by August 6). Please take a moment to read the authors’ rebuttal thoroughly and engage in the discussion. Ideally, every reviewer will respond so the authors know their rebuttal has been seen and considered.

Thank you for your prompt participation!

Best regards,

AC

Reviewer: This raises some concern about whether the baselines in your comparison were intentionally under-tuned to emphasize the benefit of your approach.

Thank you for raising this critical point. We take this concern very seriously and want to assure you that the baselines were tuned to ensure they can be trained normally with EDM on CIFAR-10. And we observe the item variantion under this condition. Specifically, we apply the pseudo-huber loss specified in iCT paper, which has the similar effect to the tangent normalization technique in sCM paper, and under which the loss can decrease normally. We also let it grow from $s=t$ in the first several iterations. And we keep other parts the same, for example the time sampling strategy is uniform sampling, which do not affect the overall training. In the following, we will add more experimental results ablating the time sampling strategy for a comprehensive comparison. To provide full transparency, we will upload the core source code below.

Reviewer: Moreover, while your loss curves are indeed quite smooth, we could not find a convincing theoretical explanation in Theorems 2 and 3 for such stability. These theorems primarily describe what the network learns in idealized settings. Even assuming Lipschitz continuity, this alone cannot explain such a drastic reduction in loss variance. From our practical experience, what does reduce variance in MSE-type losses is importance sampling [2], but we did not see any mention of such techniques in the relevant lines (L187–200, L257–268).

Theorems 2 and 3 work to establish the theoretical feasibility and local accuracy of our method, rather than to provide a direct proof of its variance reduction. You are also correct that importance sampling is a well-known technique for reducing loss variance, but we do not use, and it is not a focus in our paper.

The primary source of the drastic stability improvement you observed is not a sophisticated sampling strategy, but rather the inherent stability of our proposed loss target. Our targets, such as the velocity estimation $-\\boldsymbol{x}\_0 + \\boldsymbol{z}$ (identical to the diffusion target) or its conditional expectation $\\mathbb{E}(-\\boldsymbol{x}\_0 + \\boldsymbol{z}|\\boldsymbol{x}\_t)$ are analogous to those in highly stable, standard diffusion models. Our training process inherits this stability directly. In contrast, the loss target in CMs, $\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s))$, relies on a numerical derivative. This is computed via a finite difference between two noisy model evaluations, which is then divided by a small time step. This process inherently amplifies noise and leads to instability, especially early or middle in training when the model $\boldsymbol{f}_{\theta^-}$ is not yet accurate.

In the final version, we will:

1. Revise the main text (around L187–200) to explicitly articulate that the stability stems from the nature of the loss target, better connecting our theory to the empirical results.
2. Add a discussion to the related work section on variance reduction techniques, including importance sampling, to better contextualize our approach and its advantages.

We sincerely thank the reviewer for the valuable feedback and for recognizing the theoretical soundness and simplicity of our method. We will address the concerns below.

Question 1. Key Differences with MeanFlow

The core difference can be summarized succinctly: To achieve the same objective of fitting the average velocity (the secant), MeanFlow uses differentiation to construct its loss, whereas our method uses integration. With identical model parameterization, the two methods can be viewed as differential and integral counterparts.

The authors of MeanFlow notes in Section 4.1 that, “However, directly using the average velocity defined by Eq. (3) as ground truth for training a network is intractable, as it requires evaluating an integral during training.” The essential difficulty is, evaluating the integral requires a large amount of samples (i.e., a full inference-like trajectory). But in deep learning, using such many samples to evaluate the integration in one iteration is impractical. MeanFlow circumvents this by using differentiation, while we use Monte Carlo integration, where the key is to convert the many-sample evaluation to one-sample evaluation of the integral.

The above fundamental difference leads to our method's distinctive features: local accuracy in theory, target stability in principle, efficiency in application.

1. Theoretical Soundness: From a theoretical standpoint, a key feature that distinguishes our method is its ability to achieve local accuracy without requiring explicit differentiation in the loss function, as shown in Theorem 2 and Theorem 3.

2. Target Stability: Another crucial point for stable training is the target stability, i.e., the stability of the prediction target in the loss function. We apologize for not elaborating on this in the main text. For simplicity, let's compare the targets under the OT-FM interpolant ($d$ denotes some distance metric):

   • Diffusion Models (Flow Matching): $\mathcal{L}(\theta)=d(\\boldsymbol{v}\_\theta(\\boldsymbol{x}\_t,t)-(-\\boldsymbol{x}_0+\\boldsymbol{z}))$. The target is the velocity estimation $-\\boldsymbol{x}\_0+\\boldsymbol{z}$.
   • MeanFlow / CM / CTM (similar): $\mathcal{L}(\theta)=d(\\boldsymbol{f}\_\theta(\\boldsymbol{x}\_t,t,s)-((-\\boldsymbol{x}_0+\\boldsymbol{z})+(s-t)\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s))$. The target is $-\\boldsymbol{x}_0+\\boldsymbol{z} + (s - t)\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)$. The latter item $\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)$ (or the discrete version $\\frac{\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)-\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_{t-\Delta t},t-\Delta t,s)}{\Delta t}$) is highly unstable, which has been substantially studied in [1].
   • Our Method: The targets are either the velocity estimation $(-\\boldsymbol{x}\_0+\\boldsymbol{z})$ or the velocity $\\boldsymbol{v}(\\boldsymbol{x}_t,t)$ itself.

   Our method is the only one among these to feature a target with stability comparable to that of standard diffusion models. This inherent stability of the prediction target helps explain the robust training behavior we observe. To see the practical target stability, we also conduct an experiment with EDM on CIFAR-10, where we track the specific items in the loss functions, and record the standard deviation in the following table. The statistics shown in the table clearly support the stability of our method.

   Loss (Inspected item)/Training iteration	10	20	30	40	50	100	200	300	400	500
   CM ($\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s))$)	0.2627	15.4339	30.9154	40.7293	79.1750	101.8844	97.2164	134.4441	72.3149	97.4782
   CM ($\\frac{\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)-\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_{t-\Delta t},t-\Delta t,s)}{\Delta t}$)	0.2810	10.1618	34.4472	88.6847	39.1018	59.5919	143.4175	62.3149	69.5990	137.7932
   SDEI ($\\boldsymbol{v}(\\boldsymbol{x}_r,r)$)	3.1713	3.0427	4.1342	3.4504	3.3060	2.6381	3.2294	3.5089	3.4080	3.2673
   STEI ($\\boldsymbol{f}_{\theta^-}(\\boldsymbol{x}_r,r,r)$)	4.4030	4.2680	4.1115	3.8282	4.3783	4.5072	4.3712	4.0905	3.9820	4.0967
   SDEE ($\\boldsymbol{v}(\\boldsymbol{x}_r,r)$)	2.8809	3.2181	3.5562	3.0221	3.7394	3.0247	3.8801	3.2247	4.0112	3.5830
   STEE ($-\\text{sin}(t)\\boldsymbol{x}_0+\\text{cos}(t)\\boldsymbol{z}$)	4.2049	4.1783	4.6112	4.0290	4.6454	4.5360	4.9838	4.9387	5.0872	4.3094

3. Training Efficiency (Speed and Memory): In practical application, methods like continuous-time CM and MeanFlow require Jacobian-vector product (JVP) operations to compute the loss. This imposes a significant computational burden, especially in PyTorch (while the MeanFlow authors use JAX, PyTorch remains the more widely adopted framework). To provide a direct comparison, we benchmarked training speed and memory usage under the same EDM setup (the batch size is 512 on 8 A100 GPUs with 64 per GPU) on CIFAR-10 as in our main text.

   Method	Training Speed (sec/kimg)	Memory Usage (Per GPU, GB)
   Diffusion Model (Teacher)	0.57	8.69
   MeanFlow	1.04	38.73
   SDEI	0.91	9.64
   STEI	1.42	16.95
   SDEE	0.91	9.64
   STEI	0.72	9.34

   As the table shows, our method's training overhead is modest. For example, STEI adds only 26% to the teacher model's training time, whereas MeanFlow adds over 82%. The difference in memory consumption is even more stark: STEI increases usage by only 7%, while MeanFlow requires a 346% increase. And the total training duration required of our method is much less. For example, both fine-tuning from existing teacher models, our method converges with only 50M images on CIFAR-10, whereas sCM requires 200M.

Question 2. Scalability and Applicability

We would like to clarify that our method, like MeanFlow and Consistency Models, is also based on the exact solution of the diffusion ODE and learns the same average velocity field. Our approach does not straighten or alter the ODE trajectory itself, which is a technique used by other methods like 2-Rectified Flow [2]. Therefore, there is no inherent difference in applicability between our method and MeanFlow. Our method is designed for, and has been demonstrated to work effectively in both distillation and training settings. Regarding scalability, we refer to Section 5.3 in the main text, where we scale the model size. The results show that performance consistently improves with a larger model, which can be a supporting evidence for scalability on model size. 

Question 3. One-step Generation Performance

We provide more 1- and 2-step results in the table below.

As explained in Question 2, akin to MeanFlow, our method also learns the average velocity. For explanation of the weaker one-step generation, we can see in Theorem 2 and Theorem 3 in the main text, our loss functions emphasize on local accuracy without explicit differentiation. Our loss function is most accurate over shorter integration intervals, and its performance naturally degrades as the step interval becomes very large. We would like to highlight that it is a trade-off. Methods like CM, MeanFlow optimize for global accuracy but do so by introducing either discretization errors or explicit derivative terms in their loss functions, which leads to potential training instability. Our method's "sweet spot" is in the modest few-step regime (e.g., 4 or 8 steps). For scenarios where training speed, memory efficiency, and stability are critical—especially with large models or limited resources—and a modest number of steps is acceptable, our approach provides a highly practical and efficient solution. We acknowledge this limitation and note the performance gap between 1-step and 8-step generation in the paper.

Question 4. Clarity on Method Variants

The naming convention is based on two key design choices, estimation strategy and training mode. According to Figure 1, the main goal is to learn the secant function $\\boldsymbol{f}(\\boldsymbol{x}_t,t,s)$ such that we can jump from $\\boldsymbol{x}_t$ to $\\boldsymbol{x}_s$ at inference. First, our loss target is set as a random tangent (the green line in the figure). To apply Monte Carlo integration and Picard iteration, we should derive an end point $\\boldsymbol{x}_t$ and a uniformly selected interior point $\\boldsymbol{x}_r$. Considering that we can only access to one of these two at each training iteration, we propose to use the model itself to estimate either the interior point from the true end point, or the end point from the true interior point. We denote the former as “EI”, i.e., “estimating the interior point”, while name the latter as “EE”, i.e., “estimating the end point”. Furthermore, since the target can either be a pretrained diffusion model, or be estimated during training, both “EI” and “EE” include a distillation (D) version and a training (T) version. At the front, we add an “S” denoting “secant losses”, leading to our total four variants: SDEI, STEI, SDEE and STEE.

[1] Simplifying, Stabilizing and Scaling Continuous-time Consistency Models. Cheng Lu and Yang Song. ICLR 2025

[2] Flow straight and fast: Learning to generate and transfer data with rectified flow. Xingchao Liu et al. ICLR 2023

Dear Reviewer,

This is a gentle reminder that the Author–Reviewer Discussion phase ends within just three days (by August 6). Please take a moment to read the authors’ rebuttal thoroughly and engage in the discussion. Ideally, every reviewer will respond so the authors know their rebuttal has been seen and considered.

Thank you for your prompt participation!

Best regards,

AC

We sincerely thank the reviewer for the valuable and constructive feedback. We are also grateful for the recognition of our work's originality and clarity. We will address the reviewer's concerns below.

Regarding Weakness 1

Reviewer: Consistency models, or any model that does any-size jumps (like MeanFlow, shortcut models), can be understood as a gradual transition from tangents to secants. The contribution of the work is not really a framework or the contrast between tangents and secants, but rather a new way of training these any-step predictors.

We agree with the reviewer's assessment. Our core contribution is indeed the development of the secant expectation algorithms for training few-step, ODE-based generative models. The discussion of the relationship between tangents and secants is intended to provide a clearer geometric intuition (like that in MeanFlow paper) and a foundational basis for our model parametrization and loss construction, rather than being the central contribution of this paper.

Regarding Weakness 2

Reviewer: Since the contribution of the paper is really a new way of training an existing family of models, we need to compare their empirical performances. It seems like in almost all cases, it lags behind IMM without the help of guidance intervals. The scores on CIFAR are not really convincing either, given the competition. And if we include the comparisons with MeanFlow, the presented results do not inspire confidence in the proposed method. [...] why use the proposed losses over others?

We thank the reviewer for raising this important point. We compare the mentioned methods as follows.

Comparison with IMM: We wish to highlight a fundamental conceptual distinction between the two methods, which places them in different categories of few-step generative models. IMM adopts distribution-level loss—the Maximum Mean Discrepancy (MMD) loss. In contrast, our approach is particle-based and grounded in finding the exact solution of the ODE. And our contribution lies in developing a stable and efficient new method for particle-based few-step diffusions. In this context, our work has its unique advantages compared to other related methods as follows.

Comparison with MeanFlow, Consistency Models (CM), Consistency Trajectory Models (CTM), etc.: We wish to highlight three key advantages of our approach: local accuracy in theory, target stability in principle, efficiency in application.

1. Theoretical Soundness: From a theoretical standpoint, a key feature that distinguishes our method is its ability to achieve local accuracy without requiring explicit differentiation in the loss function, as shown in Theorem 2 and Theorem 3.

2. Target Stability: Another crucial point for stable training is the target stability, i.e., the stability of the prediction target in the loss function. We apologize for not elaborating on this in the main text. For simplicity, let's compare the targets under the OT-FM interpolant ($d$ denotes some distance metric):

   • Diffusion Models (Flow Matching): $\mathcal{L}(\theta)=d(\\boldsymbol{v}\_\theta(\\boldsymbol{x}\_t,t)-(-\\boldsymbol{x}_0+\\boldsymbol{z}))$. The target is the velocity estimation $-\\boldsymbol{x}\_0+\\boldsymbol{z}$.
   • MeanFlow / CM / CTM (similar): $\mathcal{L}(\theta)=d(\\boldsymbol{f}\_\theta(\\boldsymbol{x}\_t,t,s)-((-\\boldsymbol{x}_0+\\boldsymbol{z})+(s-t)\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s))$. The target is $-\\boldsymbol{x}_0+\\boldsymbol{z} + (s - t)\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)$. The latter item $\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)$ (or the discrete version $\\frac{\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)-\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_{t-\Delta t},t-\Delta t,s)}{\Delta t}$) is highly unstable, which has been substantially studied in [1].
   • Our Method: The targets are either the velocity estimation $(-\\boldsymbol{x}\_0+\\boldsymbol{z})$ or the velocity $\\boldsymbol{v}(\\boldsymbol{x}_t,t)$ itself.

   Our method is the only one among these to feature a target with stability comparable to that of standard diffusion models. This inherent stability of the prediction target helps explain the robust training behavior we observe. To see the practical target stability, we also conduct an experiment with EDM on CIFAR-10, where we track the specific items in the loss functions, and record the standard deviation in the following table. The statistics shown in the table clearly support the stability of our method.

   Loss (Inspected item)/Training iteration	10	20	30	40	50	100	200	300	400	500
   CM ($\\frac{d}{dt}\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s))$)	0.2627	15.4339	30.9154	40.7293	79.1750	101.8844	97.2164	134.4441	72.3149	97.4782
   CM ($\\frac{\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_t,t,s)-\\boldsymbol{f}\_{\theta^-}(\\boldsymbol{x}\_{t-\Delta t},t-\Delta t,s)}{\Delta t}$)	0.2810	10.1618	34.4472	88.6847	39.1018	59.5919	143.4175	62.3149	69.5990	137.7932
   SDEI ($\\boldsymbol{v}(\\boldsymbol{x}_r,r)$)	3.1713	3.0427	4.1342	3.4504	3.3060	2.6381	3.2294	3.5089	3.4080	3.2673
   STEI ($\\boldsymbol{f}_{\theta^-}(\\boldsymbol{x}_r,r,r)$)	4.4030	4.2680	4.1115	3.8282	4.3783	4.5072	4.3712	4.0905	3.9820	4.0967
   SDEE ($\\boldsymbol{v}(\\boldsymbol{x}_r,r)$)	2.8809	3.2181	3.5562	3.0221	3.7394	3.0247	3.8801	3.2247	4.0112	3.5830
   STEE ($-\\text{sin}(t)\\boldsymbol{x}_0+\\text{cos}(t)\\boldsymbol{z}$)	4.2049	4.1783	4.6112	4.0290	4.6454	4.5360	4.9838	4.9387	5.0872	4.3094

3. Training Efficiency (Speed and Memory): In practical application, methods like continuous-time CM and MeanFlow require Jacobian-vector product (JVP) operations to compute the loss. This imposes a significant computational burden, especially in PyTorch (while the MeanFlow authors use JAX, PyTorch remains the more widely adopted framework). To provide a direct comparison, we benchmarked training speed and memory usage under the same EDM setup (the batch size is 512 on 8 A100 GPUs with 64 per GPU) on CIFAR-10 as in our main text.

   Method	Training Speed (sec/kimg)	Memory Usage (Per GPU, GB)
   Diffusion Model (Teacher)	0.57	8.69
   MeanFlow	1.04	38.73
   SDEI	0.91	9.64
   STEI	1.42	16.95
   SDEE	0.91	9.64
   STEI	0.72	9.34

   As the table shows, our method's training overhead is modest. For example, STEI adds only 26% to the teacher model's training time, whereas MeanFlow adds over 82%. The difference in memory consumption is even more stark: STEI increases usage by only 7%, while MeanFlow requires a 346% increase. And the total training duration required of our method is much less. For example, both fine-tuning from existing teacher models, our method converges with only 50M images on CIFAR-10, whereas sCM requires 200M.

In summary, while our method may not achieve state-of-the-art FID scores, it presents a compelling trade-off. For scenarios where training speed, memory efficiency, and stability are critical—especially when working with large models or limited computational resources, and modest step numbers like 4 or 8 are acceptable, our approach provides a highly practical and efficient solution.

Regarding Question 1

Reviewer: I do not quite understand the significance of the sampling of $r$. The theory demands a uniform weighted average right? So even if one utilizes a different sampling distribution, they need to rebalance via importance sampling.

Mathematically, a uniform weighted average and a non-uniform distribution rebalanced with importance sampling are equivalent. However, in the context of deep learning, the choice of sampling distribution corresponds to how we allocate the training effort across different time intervals. We apologize for not clarifying this motivation in the main text. Inspired by recent findings in the diffusion model literature (e.g., EDM[2], SD3[3], FasterDiT[4]), where focusing training efforts on specific time intervals was shown to accelerate convergence, we decide to explore if a similar strategy would benefit our method. We experiment with different distributions for sampling the time points $r$. However, our empirical results indicate that allocating training effort uniformly across all intervals works best.

Regarding Question 2

Reviewer: The end-point version of the training requires the model to be able to invert (from data to noise), which could be a waste of model capacity.

The end-point variant of our training algorithm indeed requires the model to learn the inversion from data back to noise, which could command additional model capacity compared to the intermediate-point variant. And here we would like to emphasize that there is a trade-off between model capacity and training computation. As shown in Table 1 in the main text, the benefit of this approach is a more efficient training step: while STEE may use more model capacity, it saves two forward pass evaluations and one backpropagation step per iteration compared to STEI.

[1] Simplifying, Stabilizing and Scaling Continuous-time Consistency Models. Cheng Lu and Yang Song. ICLR 2025

[2] Elucidating the design space of diffusion-based generative models. Tero Karras et al. NeurIPS 2022

[3] Scaling rectified flow transformers for high-resolution image synthesis. Patrick Esser et al. ICML 2024

[4] FasterDiT: Towards faster diffusion transformers training without architecture modification. Jingfeng Yao et al. NeurIPS 2024