FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation

We thank the reviewer for their recognition of the training-free design and the effectiveness of our patch-based variance reduction approach.

Runtime speedup

In response to the reviewer’s feedback, we made small implementation changes that preserve correctness. Specifically, we replaced redundant operations and removed unnecessary computation (e.g., replacing loops over tensors with built-in PyTorch functions). The code changes did not change the results – we verified on all samples from our project’s website that the obtained outputs are bit-exact identical before and after the change. These optimizations to a x1.29 speedup, reducing the mean runtime from 234.30 seconds to 180.67 seconds. FlowMo now incurs an overhead of only x1.82 (down from x2.39) compared to baseline.

Adding FlowMo during training.

Thank you for the insightful suggestion. First, we note that integrating our refinement into training is a promising direction for future research, since prior works like [1] show that inference-time objectives (e.g., [2]) can be effectively adapted into training, and obtain similar gains. Since FlowMo is similar in spirit to these image-based inference-time optimizations, we believe it will yield similar benefits when applied to pretraining or fine tuning.
However, attempting this is infeasible under the limits of our current computational resources; we only have access to two H100 GPUs, and backpropagating through larger batches with latent gradients would be significantly more demanding than our inference-time setup.

We appreciate your time and constructive feedback. We would be happy to address any further questions or concerns you may have during the discussion period.

[1] Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models

[2] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

We thank the reviewer for highlighting the method’s generalizability across models and its versatility as a training-free, plug-and-play refinement.

Trade-off between temporal consistency and dynamic degree

Please note that this is addressed directly in both the motivational experiments and the main results of the paper, showing that FlowMo improves temporal coherence without meaningfully compromising motion dynamics.

1. In our motivation (Sec. 3.2, L. 144–152), we specifically analyze high-motion videos and find that patch-wise variance correlates with coherence even when motion is high, indicating our loss targets coherence, not reduced dynamics.
2. In our main results (Sec. 4.2, App. A.2, Table 2), we show that FlowMo reduces Dynamic Degree by less than 1.5% across both models, a negligible drop given the overwhelming benefit to the overall Quality Score (Wan2.1: +8.0%, CogVideoX: +11.25%). In contrast, FlowMo’s closest baseline, FreeInit, causes a 16.2% drop in Dynamic Degree (see L. 568-574, Appendix A). This is a further indication that our method does not incur a significant drop in dynamic degree, even compared to its closest baselines.
3. Finally, note that Dynamic Degree reflects motion magnitude, not coherence; for instance, completely incoherent motion (e.g. flickering) can inflate Dynamic Degree scores. Thus, slight reductions in this score are expected when coherence improves.

Therefore, rather than exploiting a trade-off between the two measures, FlowMo improves coherence without compromising dynamic degree.

To provide a clear overview, we briefly summarize the core contributions of our work:

• We present a systematic analysis of motion in video diffusion models (Sec. 3.2), revealing that motion often emerges in later denoising steps, distinct from appearance features. We justify our insights both quantitatively and qualitatively. We further show that patch-wise variance is a strong predictor of temporal coherence, even in high-motion videos (Fig. 2, L. 144-154).
• We propose FlowMo, a training-free, inference-time method that significantly improves temporal coherence with minimal impact on motion dynamics. It is lightweight, plug-and-play, and requires no retraining.
• We demonstrate strong cross-model generalization across Wan2.1 and CogVideoX (which differ in architecture, training data, and base performance) highlighting the robustness and broad applicability of our approach.

Justification of design choices.

Due to the character limit, we are unable to include the full response here. We kindly refer the reviewer to our response to Reviewer gJqQ, where we provide a theoretical justification for using the maximum operator in FlowMo’s loss. That response also includes both empirical and theoretical justifications for the choice of the 1–12 timestep range.

Furthermore, the table below shows a quantitative comparison of VBench metrics between the WAN2.1-1.3B baseline, FlowMo, and the following ablations: Mean (i.e., regulating mean patch-wise variance of the debiased latent, as opposed to max), W/O Debias (i.e., regulating max patch-wise variance of the un-debiased latent), and Steps 0-30 (applying optimization over the first 30 steps, instead of all 50, due to computational and time constraints).

In accordance with the qualitative results presented in the paper (Fig. 6), FlowMo demonstrates the best Motion Smoothness, with roughly 50% more improvement over the best ablation. It also achieves the strongest performance across the three aggregated metrics: +5.86% on Semantic Score, +6.07% on Quality Score, and +5.03% on Final Score compared to the closest ablation. As for Dynamic Degree, applying optimization over 30 steps leads to a significant drop, whereas the other ablations—and FlowMo itself—show no meaningful degradation, indicating that motion magnitude is largely preserved.

The quantitative metrics are consistent with our motivation and theoretical justification. First, replacing the maximum operation with a mean significantly weakens the effect of the optimization, as was also argued theoretically above. Removing the debiasing operator results in a comparable degradation, which aligns with prior observations that text-to-video models tend to prioritize appearance features, which can overshadow motion in the loss signal (e.g., [2]). Lastly, applying the optimization across the first 30 denoising steps leads to a clear performance drop. This is because the coarse denoising steps primarily determine motion, structure, and overall layout. In later steps, these aspects are already fixed, and the loss, although still valid, can only influence high-frequency details. As a result, applying the optimization at later stages tends to introduce artifacts rather than improve motion quality. This observation aligns with findings from prior works (e.g., [1], [3]), which similarly emphasize the importance of applying optimization during the early denoising stages.

VBench results on Wan2.1

While the official VBench score for Wan2.1-1.3B is reported as 84.26%, the exact inference settings (e.g., resolution, prompt formulation) used to obtain that result have not been publicly detailed. We suspect the performance gap is due to our use of 480p resolution, which is lower than the model’s maximum supported resolution (720p, as stated on the official website). Due to computational limitations, we were unable to run our experiments at the higher resolution. Please note that all our experiments were conducted using the publicly released code and checkpoints, and we followed the default configurations as provided in both the Wan2.1 and VBench repositories. Our goal was to ensure consistency and reproducibility, and under these conditions, FlowMo demonstrates clear and meaningful improvements.

Influence by external factors.

We would like to kindly emphasize that our experiments were carefully designed to cover all types of motion, with various complexities, prompts, and settings. Firstly, the VBench dataset includes a wide variety of prompts, ranging from simple to complex, and from static scenes to high-motion scenarios. Secondly, we enclose studies conducted on VideoJAM-bench [2] (Fig. 5,8, L. 247-253), which was carefully and specifically designed to examine motion, and includes highly complex motion types such as gymnastics, physics, rotations, etc. Across these diverse and comprehensive benchmarks, FlowMo consistently shows strong performance and improved temporal coherence, further supporting its robustness to prompt complexity and motion diversity while improving temporal consistency.

Runtime speedup

In response to the reviewer’s feedback, we made small implementation changes that preserve correctness. Specifically, we replaced redundant operations and removed unnecessary computation (e.g., replacing loops over tensors with built-in PyTorch functions). The code changes did not change the results – we verified on all samples from our project’s website that the obtained outputs are bit-exact identical before and after the change. These optimizations led to a x1.29 speedup, reducing the mean runtime from 234.30s to 180.67s. As a result, FlowMo now incurs an overhead of only x1.82 (down from x2.13) relative to the baseline.

---

Thank you once again for your helpful observations and careful review. We would be pleased to further clarify any remaining points or engage in additional discussion.

References

[1] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

[2] VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

[3] Image Generation from Contextually-Contradictory Prompts

Thank you for your continued engagement and constructive feedback. We are pleased that our responses have addressed several of your concerns, particularly regarding dynamic degree and ablation studies.

Regarding the WAN2.1 baseline performance gap:

We appreciate your recommendation to conduct experiments with higher-performing WAN baseline implementations. We want to clarify an important technical constraint we encountered following this discussion: when attempting to run WAN2.1-T2V-1.3B with the flag --size 1280*720 to match the reported official resolution, we encounter an assertion error:

"AssertionError: Unsupport size 1280*720 for task t2v-1.3B, supported sizes are: 480*832, 832*480."

This technical limitation is also documented in the official WAN2.1 GitHub repository, which states:

"The 1.3B model is capable of generating videos at 720P resolution. However, due to limited training at this resolution, the results are generally less stable compared to 480P."

Additionally, the HuggingFace model card confirms that this specific resolution is not supported in the published release.

Given these constraints, our experiments were conducted at the maximum practically supported resolution (480*832) using the publicly available codebase. While this may contribute to the performance gap you noted, it ensures our results are reproducible and based on the actual capabilities of the released model. Importantly, the benchmark motion metrics that we report are similar to the official WAN2.1 motion metrics, and these are the most relevant to our paper's focus on temporal coherence improvements. Additionally, we would like to highlight that FlowMo improves motion-related metrics even when compared to the published results obtained at higher resolution. Specifically, we achieve:

• 98.56% vs 97.44% in Motion Smoothness
• 27.30% vs 25.32% in Temporal Style

Our method's effectiveness is demonstrated consistently across both WAN2.1 and CogVideoX models, with the latter showing even stronger improvements (+11.25% vs +8.0% Quality Score improvement), suggesting our approach's robustness extends beyond any single baseline implementation.

Regarding motion coverage:

We acknowledge and accept your feedback that our statement "our experiments were carefully designed to cover all types of motion" in the rebuttal was overly absolute. Since the rebuttal cannot be edited at this stage, we recognize this phrasing and agree it should be more measured. A more accurate characterization would be that our experiments cover a diverse range of motion types across comprehensive benchmarks (VBench and VideoJAM-bench), demonstrating consistent improvements across various motion complexities and scenarios.

We believe these clarifications, combined with our previous responses on dynamic degree preservation and ablation studies, strengthen the paper's contribution while maintaining transparency about technical limitations and scope.

We thank the reviewer for the encouraging feedback and are glad they found the motivation analysis convincing, the method simple and accessible, and the writing clear.

Runtime speedup

In response to the reviewer’s feedback, we made small implementation changes that preserve correctness. Specifically, we replaced redundant operations and removed unnecessary computation (e.g., replacing loops over tensors with built-in PyTorch functions). The code changes did not change the results – we verified on all samples from our project’s website that the obtained outputs are bit-exact identical before and after the change. These optimizations led to a x1.29 speedup, reducing the mean runtime from 234.30s to 180.67s. As a result, FlowMo now incurs an overhead of only x1.82 (down from x2.13) relative to the baseline.

Moving FreeInit to the main text.

Thank you. Following your suggestion, we will move both the quantitative and qualitative comparisons of FreeInit from the appendix into the main text.

Additional quantitative ablation results.

The table below shows a quantitative comparison of VBench metrics between the WAN2.1-1.3B baseline, FlowMo, and the following ablations: Mean (i.e., regulating mean patch-wise variance of the debiased latent, as opposed to max), W/O Debias (i.e., regulating max patch-wise variance of the un-debiased latent), and steps 0-30 (applying optimization over the first 30 steps, instead of all 50, due to computational and time constraints).

In accordance with the qualitative results presented in the paper (Fig. 6), FlowMo demonstrates the best Motion Smoothness, with roughly 50% more improvement over the best ablation. It also achieves the strongest performance across the three aggregated metrics: +5.86% on Semantic Score, +6.07% on Quality Score, and +5.03% on Final Score compared to the closest ablation. As for Dynamic Degree, applying optimization over 30 steps leads to a significant drop, whereas the other ablations—and FlowMo itself—show no meaningful degradation, indicating that motion magnitude is largely preserved.

The quantitative metrics are consistent with our motivation and theoretical justification. First, replacing the maximum operation with a mean significantly weakens the effect of the optimization, as was also argued theoretically above. Removing the debiasing operator results in a comparable degradation, which aligns with prior observations that text-to-video models tend to prioritize appearance features, which can overshadow motion in the loss signal (e.g., [4]). Lastly, applying the optimization across the first 30 denoising steps leads to a clear performance drop. This is because the coarse denoising steps primarily determine motion, structure, and overall layout. In later steps, these aspects are already fixed, and the loss, although still valid, can only influence high-frequency details. As a result, applying the optimization at later stages tends to introduce artifacts rather than improve motion quality. This observation aligns with findings from prior works (e.g., [1], [3]), which similarly emphasize the importance of applying optimization during the early denoising stages.

Inconsistency with CFG Definition.

Thank you for pointing this out. We will revise line 3 In Algorithm 1 to match the correct CFG structure: $u_{\theta, t_i} \gets u_{\theta, t_i \mid \emptyset} + \rho \cdot (u_{\theta, t_i \mid P} - u_{\theta, t_i \mid \emptyset} )$

Cost-effectiveness of FlowMo and its proper applications.

FlowMo is designed to be cost-effective: it requires no additional training and introduces only a modest runtime overhead (which is now x1.82) during inference. This makes it practical for applications where motion quality is critical, such as content creation or video refinement, while preserving efficiency.

Additional training-free baselines

In addition to the training-free methods that we have included in our study, the only relevant method that we could not include is VideoGuide. This is due to the incompatible architecture, since this method was developed for a U-net based architecture and is not easily adaptable (see lines 96-103).

We are happy to conduct further comparisons upon request.

Conduct a careful writing check.

We will conduct a thorough review of the writing to improve clarity and resolve any ambiguities in the final version of the paper.

---

We sincerely thank you for your insights and feedback. We would be glad to answer any follow-up questions and continue the discussion.

References

[1] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

[2] FreeInit: Bridging Initialization Gap in Video Diffusion Models

[3] Image Generation from Contextually-Contradictory Prompts

[4] VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

We thank the reviewer for the thoughtful feedback and for recognizing our core insights, design, and evaluation.

Runtime speedup

In response to the reviewer’s feedback, we made small implementation changes that preserve correctness. Specifically, we replaced redundant operations and removed unnecessary computation (e.g., replacing loops over tensors with built-in PyTorch functions). The code changes did not change the results – we verified on all samples from our project’s website that the obtained outputs are bit-exact identical before and after the change. These optimizations led to a x1.29 speedup, reducing the mean runtime from 234.30s to 180.67s. As a result, FlowMo now incurs an overhead of only x1.82 (down from x2.13) relative to the baseline.

The choice of maximal patch wise variance

We provide both intuitive and theoretical justifications for the choice of the max operator in our optimization.

Intuitively, artifacts are typically a local phenomenon that is dominated by specific spatiotemporal locations in which the patch statistics are abnormal (compared to what exists in the training set). Using the mean operator dilutes the impact of these anomalies, since it is dominated by the majority of the patches. However, their presence can be captured by a max operator, making it a natural choice to detect and mitigate inconsistencies.

Theoretically, the following lemmas justify optimizing the maximum patch-wise variance to keep generated latents in-distribution.

Lemma 1. Noising of real videos keeps patch-wise temporal variance bounded by $1 + V_{\max}$, where $V_{\max}$ is a dataset-dependent constant that bounds patch-wise variance for all patches of all videos in the training data.

First, note that $V_{\max}$ exists since the model was trained on a finite dataset of real-valued latents. Furthermore, assuming the dataset consists of natural videos, and since VAEs minimize latent variance, $V_{\max}$ is likely to be relatively small.

Now, recall that noising a latent of a real video $z_0$ to timestep $t$ is done by:

$z_t = \sqrt{\alpha_t} z_0 + \sqrt{1 - \alpha_t} \epsilon \quad \text{where } \epsilon \sim \mathcal{N}(0, I)$

Thus, the temporal variance of each patch $ij$ is given by:

$V(z_t[ij]) = \alpha_t V(z_0[ij]) + (1 - \alpha_t) = 1 + \alpha_t (V(z_0[ij]) - 1) \leq 1 + V_{\max}$

Hence the maximum patch-wise variance of all samples during training is bounded.

Lemma 2. Consider the denoising process starting from pure Gaussian noise. Suppose that in early timesteps (i.e., when $\alpha_t$ is small), each patch of the denoised latent remains approximately Gaussian but with unknown variance due to guidance, decoder mismatch, or score error, i.e., $z_{k, i, j} \sim \mathcal{N}(0, s_{ij})$, with $s_{ij}$ varying across space. Then, the maximal patch-wise variance is $\Omega\left( \frac{\log(W \cdot H)}{F} \cdot \max s_{ij} \right)$ w.h.p., where $F,W,H$ are the number of frames, width, and height of the latent, respectively.

We note that the assumptions in Lemma 2 are empirically satisfied in early denoising timesteps: the denoised latent remains approximately Gaussian, and patch-wise variances exhibit significant spatial variability consistent with the stated form.

Intuitively, the lemma holds because at high noise levels, the per-pixel variance distribution is heavy-tailed, so outliers are likely to occur. Formally, given the assumptions in the lemma, the temporal variance of each patch is distributed $V(z_t[ij]) \sim \frac{s_{ij}}{F - 1} \cdot \chi_{F - 1}^2$, where $F$ is the number of latent frames. The maximal patch-wise variance is thus the maximum over such $\chi^2$ distributions, which lies in the Gumble domain and is known to satisfy the property in the lemma.

Hence, as the number of patches $W \cdot H$ grows, local outliers (i.e., patches with high patch-wise variance) appear with non-negligible probability.

Lemma 3. A global stability condition for the numerical integration step in the denoising process requires that $\eta_t \Vert J_{\max} \Vert \sqrt{(F-1) s_{\max}} \leq C$, where $J_{ij}=\frac{\partial \epsilon_\theta}{\partial Z_{ij}}$ is the Jacobian at patch $ij$, $s_{ij}$ is its patch-wise variance, and $C$ is the Courant number.

Let $z_{t-h} \approx z_t - \eta_t JZ$ be a linearization of the denoising step.

A stability condition for solving the ODE numerically is the CFL condition, which requires every patch $ij$ to satisfy $\eta_t \Vert J_{ij} \Vert \leq C$. Otherwise, the numerical process could be expensive and not converge. If the CFL condition holds, the local step magnitude satisfies:

$\Vert z_{t-h}[ij] - z_t[ij] \Vert \leq \eta_t \Vert J_{ij} \Vert \Vert z_{ij} - \bar{z}_{ij} \Vert$

$= \eta_t \Vert J_{ij} \Vert \sqrt{(f-1) s_{ij}}$

Therefore, a global stability condition is as mentioned in the lemma. If a single patch attains a large $s_{ij}$, the product above can cross the stability boundary, causing a local overshoot that pushes that region off the training manifold.

Corollary. In training, patch-wise variance is bounded. However, when denoising pure Gaussian noise, high patch-wise variance is a likely local phenomenon, which may drive generation out-of-distribution. Minimizing the maximum patch-wise variance is thus a practical strategy to improve ODE stability and ensure generated latents $z_t$ remain in-distribution.

Selection of timestamps 1-12

We justify the selection of timesteps 1-12 on both empirical and theoretical grounds.

We break down the question into two: why we started optimizing from the first timestep, and why we stopped after 12. First, notice that Lemma 2 above shows that large patch-wise variance, a precursor to temporal incoherence, can arise as soon as the latent remains close to Gaussian—i.e., from the very first timestep—so optimization should start as early as the first step.

As to why we stopped after 12 steps, section 3.2 shows, qualitatively and quantitatively, that coarse structure and global motion are set within the first dozen denoising steps. Figure 3 further indicates that these high-noise, “coarse” steps largely determine overall motion, making them optimal to apply FlowMo. This mirrors findings in image-based works (e.g., [1], [3]) which likewise identify early steps as critical for establishing coarse features. Furthermore, the same 1-12 range improves motion coherence on two very different models—Wan2.1 and CogVideoX (Secs. 4.1 – 4.2)—demonstrating robustness across architectures and datasets.

Adaptive timestep selection

Thank you for this valuable suggestion. First, we note that the use of a fixed intervention range is the standard practice in all other inference-time techniques we are aware of (e.g., [1], [2]). Nonetheless, adaptive selection is a promising direction for improving both runtime and output quality. In response to the review, we implemented a simple adaptive intervention strategy: if the maximum variance at a given step fell below 60 times the mean variance, we stopped the intervention early. The threshold of 60 was chosen based on the typical ratio observed at step 12 after a full 12-step intervention. This adaptive approach led to early stopping in most cases—typically between steps 8 and 10—and reduced average runtime by 20%. While performance metrics were slightly lower than in the fixed 12-step setup, we believe this initial result shows promise, and expect that more refined adaptive strategies could yield better results with relatively little effort.

Additionally, initial experiments on a subset of the data suggest that interventions applied between 8 and 15 frames result in metric values within 1% of those from the fixed setup—substantially better than the 30-frame setting (see ablations in our response to reviewer MYjg or the no-intervention baseline).

Efficient optimization strategies (e.g., fewer gradient steps, approximate gradients)

Following the review, we implemented approximate gradients (finite differences and random directional derivatives), but found that these lead to a substantial drop in video quality. Developing more efficient optimization techniques without compromising quality could be an interesting direction for future work.

Regarding the number of gradient steps: please note that FlowMo uses a single gradient step (L192–200).

Regarding implementation efficiency, please see above.

---

We would be happy to engage further and clarify any additional questions.

References

[1] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

[2] FreeInit: Bridging Initialization Gap in Video Diffusion Models

[3] Image Generation from Contextually-Contradictory Prompts