Pyramidal Flow Matching for Efficient Video Generative Modeling

We sincerely appreciate the reviewer for recognizing our work and providing valuable feedback. Below are our responses to the raised concerns.

[W1] Mathematical notations.

Sorry for the unclear mathematical notation. We have revised the draft and included the detailed explanations for the used notations in Table 3 of the appendix.

[W2] Writing style.

• Thanks for your valuable suggestions. Due to the limited time in the rebuttal phase, we will try to make the presentation of the methods section clearer in the future. We would appreciate it if you have any other detailed revision advice.

[W3/Q3] Ablation for the number of pyramid stages.

• Following your suggestion, we perform an additional ablation study on image generation (evaluated on MSCOCO) to investigate the influence of pyramid stage numbers by training the model from scratch. As shown in the following Table, using 4 stages can further improve the model's convergence speed. More pyramid stages are not being experimented since the resolution could not be evenly divided.

  FID at step	3 Pyramid stages	4 Pyramid stages
  10k	53.57	52.83
  20k	46.82	43.94
  30k	42.01	43.70
  40k	41.46	40.30
  50k	39.71	38.44

• In the paper, we set it to 3 to fully utilize low-resolution video data. For example, the videos from WebVid-10M cannot be downsampled 4 times by $2^4$ (since they are already spatially compressed by 8 using the VAE and require a patch size of 2).

[W4/Q4] Comparison of model size and FLOPs.

• The number of parameters and FLOPs are shown in the following table. While our method has a comparable model size to the open-source baselines ($\approx$ 2B), it surpasses them on VBench or EvalCrafter (Tables 1 and 2 in Section 4.3). In terms of FLOPS, we compare with CogVideoX by evaluating the computational FLOPs of each sampling step during generation. It shows that our autoregressive model yields lower FLOPs than the diffusion-based CogVideoX, thanks to its temporal compression designs.

  Model	#Parameters	FLOPs (G)	Output video	Speed (sec)
  ModelScope	1.7B	-	-	-
  LaVie	3B	-	-	-
  Open-Sora Plan v1.3	2.7B	-	-	-
  Open-Sora 1.2	1.1B	-	-	-
  CogVideoX-2B	2B	47227	720 x 480 x 49	90
  CogVideoX-5B	5B	169192	720 x 480 x 49	180
  CogVideoX 1.5-5B	5B	-	1360 x 768 x 81	1000
  Ours (384p 5s)	2B	30154	640 x 384 x 121	62
  Ours (768p 5s)	2B	112386	1280 x 768 x 121	336

• Note that the main contribution of this paper is to improve training efficiency, as highlighted in Section 4.2. Any other gains in inference efficiency are essentially a byproduct of the careful compression designs.

[Q1/Q2] Comparison of training data

• We summarize the training data used by open-source baselines in the table below. As shown, the baselines are often trained on larger video datasets than ours. Meanwhile, our model outperforms them on Vbench or EvalCrafter (Tables 1 and 2 in Section 4.3). This confirms the data efficiency of our approach.

  Model	#Videos	Video source	Overlap with ours
  ModelScope, Show-1, VideoCrafter2	10M	WebVid-10M	10M
  LaVie	25M	Vimeo25M	$\approx$ 0
  Open-Sora Plan v1.3	19M	Panda-70M	$\approx$ 1M
  Open-Sora 1.2	30M	WebVid-10M, Panda-70M, etc.	$\approx$ 11M
  CogVideoX	35M	private	unknown
  Ours	12M	WebVid-10M, OpenVid-1M, etc.	-

We sincerely appreciate the time and effort you dedicated to reviewing our paper and providing valuable feedback. Below are our responses to the raised concerns.

[W1] Implication of separate optimization.

• There are two drawbacks of the previous cascaded diffusion models: (1) Each module specialize only in generation or super-resolution, neglecting the similarity between the two tasks. This leads to worse results than end-to-end training as a joint model, similar to single-task learning vs. multi-task learning, where the latter converges to global optima. (2) They require modification of the ML infrastructure, which limits their flexibility to scale. For example, their generation and super-resolution modules are typically of different sizes, making model sharding and other useful parallelism techniques difficult to apply.

[W2] Details about the number of windows and $\gamma$.

• Our proposed method does not require much manual design, including the parameter $\gamma$ and time windows. As explained in lines 250-252 of page 5, our method derives $\gamma=-1/3$ (to guarantee the positive semi-definite property of covariance matrix) as the theoretically optimal setting (for efficient renoising) and adopts it in all experiments.

• For the number of time windows, we set it to 3 because this is the upper bound supported by our data. For example, the videos from WebVid-10M cannot be downsampled 4 times by $2^4$ (given they are already spatially compressed by 8 using the VAE and require a patch size of 2). However, we expect the use of more time windows to be beneficial to further improve the training convergence. We conduct an ablation study on image generation (MSCOCO) to investigate the influence of pyramid stage numbers by training the model from scratch. As shown in the following Table, using 4 stages can further improve the convergence speed. We did not experiment with more stages since the resolution could not be evenly divided. As for the time window range setting, we simply divide them uniformly and do not use some complicated manual designs.

  FID($\downarrow$) at step	3 Pyramid stages	4 Pyramid stages
  10k	53.57	52.83
  20k	46.82	43.94
  30k	42.01	43.70
  40k	41.46	40.30
  50k	39.71	38.44

[W3] Connection of figures to the equations.

• We apologize for any confusion about Figures 2 and 3, and explain their connections to the equations below: Figure 2 illustrates the spatial pyramid flow, including its training and inference. Specifically, Figure 2a shows its training, where the flow trajectory is curated by the start and end points in Equations (9) and (10). And Figure 2b provides its inference details, in particular the renoising step in Equation (15) across different stages.
• Figure 3 illustrates the pyramidal temporal condition. Specifically, Figure 3a shows its compressed history condition as in Equation (17), while Figure 3b shows its position encoding details in lines 315-317 of page 6. We will carefully revise the figures and their captions to reflect these connections.

[Q1] Application to text-to-image generation.

• The proposed method can also be applied to text-to-image generation. The autoregressive video generation model natively generates a high-quality image as the first frame, see Figure 5 for examples. Therefore, the pyramid-flow have the text-to-image generation capability. We have recently trained a 1024px text-to-image generation model from scratch using Pyramid Flow. Even with only a few million training images, it already shows excellent visual quality, see Figure 12(a) for the generated images.

[Q2] Advantages of flow matching.

• We adopt flow matching for its flexibility in interpolating between arbitrary source and target distributions. In contrast, DDIM and other ODE-based diffusion models typically interpolate between the standard Gaussian and data distributions, which prohibits the flexible design of pyramidal flows. In addition, this work has greatly benefited from the simple parameterization and scheduler designs of flow matching, which are crucial for scalable training.

[Q1] Training-test mismatch in autoregressive generation.

• The training-test mismatch is small because the training noise is randomly sampled from [0, 1/3], which covers the test scenario where no noise is added. A similar practice is adopted in [1], where the randomly sampled training noise covers certain noise patterns for testing. We note that one can explicitly remove the training-test discrepancy by adding a conditional flag indicating the noise level [2], but we did not observe a need in our experiment.
• Nevertheless, how to add noise to the history frames remains a key design issue in autoregressive video generation. We find that using low noise leads to temporal degradation, while high noise overemphasizes the model's ability to generate, causing the model to not strictly follow the history frames. We will continue to work on this issue in autoregressive video generation.

[Q2] Details of text conditioning.

• In terms of model structure, we add text conditions according to MM-DiT [4], namely by joint attention of text and visual features as well as AdaLN. In terms of training, we implement classifier-free guidance that drops the text condition with a probability of 10%, which is known to be essential for the generation quality of diffusion models.

---

Reference:

[1] Chen, et al. Diffusion forcing: Next-token prediction meets full-sequence diffusion. NeurIPS 2024.

[2] Valevski, et al. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837.

[3] Huh, et al. The Platonic representation hypothesis. ICML 2024.

[4] Esser, et al. Scaling rectified flow Transformers for high-resolution image synthesis. ICML 2024.

We sincerely appreciate the time and effort you dedicated to reviewing our paper and providing constructive feedback. Below we clarify the raised concerns one by one.

[W1] Inference efficiency.

• Before presenting the statistics, we note that the main contribution of this work is training efficiency (see Section 4.2) rather than inference efficiency. To evaluate the latter, we compare CogVideoX on a single NVIDIA A100 GPU in terms of total FLOPs and time to generate for different video sizes. For the computational FLOPs, we report the value of one sampling step. It is shown that our autoregressive model yields lower FLOPs and inference time than the diffusion-based CogVideoX, thanks to its pyramidal compression designs.

  Model	Output video	FLOPs(G)	Speed (sec)
  CogVideoX-2B	720 x 480 x 49	47227	90
  CogVideoX-5B	720 x 480 x 49	169192	180
  Ours (384p 5s)	640 x 384 x 121	30154	62
  CogVideoX 1.5-5B	1360 x 768 x 81	-	1000
  Ours (768p 5s)	1280 x 768 x 121	112386	336

[W2] Error drifting in autoregressive generation.

• We'd like to share two interesting observations: (1) The proposed method fails after generating 241 video frames (or 31 latent frames), which is exactly the training context length. This is more similar to LLMs where the model performs well within the training context length and fails beyond. (2) Error drifting is indeed a key problem in autoregressive video generation, but it is not specific to causal attention or full attention. For example, Figure 11 shows that autoregressive models perform worse with full attention. Thus, the essence of error drifting requires further investigation as in [1, 2].

[W3] Videos from compared methods.

• Thank you for your valuable suggestion. Indeed, a comparison to the baseline videos on the project page (as in Appendix C.3) improves clarity. We will update the project page in future revisions; it is unclear if this is allowed during rebuttal.

[W4] Baseline with more training iterations.

• To clarify, since our main focus is on training efficiency, most experiments utilize a fixed training budget. We expect that given enough training iterations, all reasonable baselines will converge to similarly good performance, as suggested by the Platonic Representation Hypothesis [3]. However, since training computation is still the performance bottleneck in most scenarios, it is important to investigate training efficiency, as in our work.

[W5] Inferior text-video alignment.

• The reason behind inferior text-video alignment is detailed in Section C.1, mainly due to the data issue:

  This is largely due to our video captioning procedure based on video LLMs which tends to produce coarse-grained captions, thus dampening these abilities.

  To illustrate this, below are 5 captions sampled from the recaptioned WebVid-10M dataset, which are significantly coarser than the baselines such as CogVideoX and Open-Sora, resulting in inferior text-video alignment. Therefore, we believe that well-captioned video datasets are critical for the development of better video generative models.

  a bunch of green grapes on a black background

  a dust storm in a city, with buildings barely visible through the sandy air

  an arieal view of a dock with a large ship and a smaller boat

  a man playing a guitar on stage at a concert

  a truck driving on a road in a desert environment

• We have recently trained the pyramid flow from scratch on the same data using the FLUX structure. During training, we filter out the low quality image-text pairs in the LAION dataset. The performance of the new pyramid-flow-miniflux model on VBench is reported in the following table. We find that improving the quality of the captions can significantly improve the text-video alignment even with fewer training iterations.

  Model	Total Score	Quality Score	Semantic Score
  Open-Sora Plan v1.1	78.00	80.91	66.38
  Open-Sora 1.2	79.76	81.35	73.39
  VideoCrafter2	80.44	82.20	73.42
  T2V-Turbo	81.01	82.57	74.76
  CogVideoX-2B	80.91	82.18	75.83
  Ours (SD3)	81.72	84.74	69.62
  Ours (Miniflux)	81.77	83.82	73.56

We sincerely appreciate the reviewer for recognizing our work and providing valuable feedback. Below are our responses to the raised concerns in your review.

[W1] Details of time windows.

• The renoising step ($e_{k+1}\neq s_k$) is derived assuming Equations (7) and (8), see Appendix A for derivation. On the other hand, as you suggested, a flow model with $e_{k+1}= s_k$ can be defined instead by the following endpoints:

  • End: $\hat x_{e_k}=e_k\mathit{Down}( x_{1},{2^k})+(1-e_k)n$,
  • Start: $\hat x_{s_k}=\mathit{Up}(s_k\mathit{Down}( x_{1},{2^{k+1}})+(1-s_k)n)$

  In our early experiments, this flow model shows inferior visual quality. We suspect this is because it cannot perform super-resolution and denoising at the same time, since it implies super-resolution first and then denoising. Overall, we recommend using Equations (7) and (8), which results in $e_{k+1}\neq s_k$.

• To avoid ambiguity in overlapping timesteps, we do not compute the timestep embedding based on the original timestep $t$, but on a globally normalized timestep $\frac{t-s_i+\sum_{k>i}(e_k-s_k)}{\sum_k(e_k-s_k)}$, where $i$ is the current pyramid stage.

• The starting point $s_K$ is indeed zero, corresponding to pure noise. The time windows are simply divided with uniformly spaced endpoints, namely $e_k=1-\frac{k}{K}$. According to Equation (26), these time windows are $[s_k,e_k]=[\frac{K-k-1}{K+k+1},1-\frac{k}{K}]$.

[W2] Ambiguity in temporal condition.

• We apologize for any confusion between Figure 1b and Figure 3. They both illustrate the temporal condition $\{x^0,\ldots,x^{i-1}\}$ that gradually changes in resolution, but Figure 3 additionally shows the current prediction $x^i$. We will revise the figure captions to make this difference clearer.
• In terms of resolution, your observation based on Figure 3 is correct, namely that $x^{i-1}$ has the same spatial resolution as $x^i$. This is because a latent of the same resolution is necessary to provide visual detail to maintain temporal consistency.

[W3] Autoregressive generation length.

• Our current model cannot generate videos of arbitrary length because it does not utilize sliding windows. Sliding window is a common technique to autoregressively generate longer videos at test time [1, 2], but in this work we focus on training models natively on long videos, rather than adding their support post hoc. Nevertheless, it is possible to combine our 10-second model with sliding windows to generate longer videos.

[W4] Quantitative results for full-sequence diffusion.

• Thanks for your valuable suggestions! We have quantitatively evaluated the FVD metric of the full-sequence diffusion baseline and our pyramid-flow on the MSR-VTT benchmark. The FVD plot along training iterations is illustrated in Figure 12(b) in the appendix. For convenience, the detailed results are also presented in the following Table. As observed, the convergence rate of pyramid-flow is significantly improved compared to the standard full-sequence diffusion.

  FVD($\downarrow$) at step	full-seq diffusion	Pyramid-Flow (ours)
  10k	513.42	355.16
  20k	450.46	315.18
  30k	403.29	277.57
  40k	370.25	209.49
  50k	310.47	165.52

---

Reference:

[1] Chen, et al. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392.

[2] Valevski, et al. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837.

[W5-1] FVD metric.

• Thank you for your constructive comments. The following table compares our model with previous baselines on the FVD metric on MSR-VTT, which is a commonly used benchmark to evaluate video generation performance. The results show that our model outperforms the competing baselines, demonstrating the effectiveness of pyramid flow.

  Model	FVD on MSR-VTT ($\downarrow$)
  CogVideo	1294
  VideoComposer	580
  VideoPoet	213
  Video-LaVIT [2]	188.36
  Ours	142.83

[W5-2] Inference and training speed.

• We first describe the training speed, which is the main contribution of our work. This has been validated by comparison with the open source baseline (Section 4.2) and by ablation studies (Figures 7 and 8 in Section 4.4). Below is a detailed summary of training costs, where our method shows a significant improvement in training efficiency:

  Model	Output video	GPU hours
  Open-Sora Plan v1.2	1280 x 720 x 93	37.8k H100 + 4.8k Ascend
  Open-Sora 1.2	1280 x 720 x 102	35k H100
  Ours (768p 10s)	1280 x 768 x 241	20.7k A100

• Next, we compare the video generation inference time and FLOPs on a single NVIDIA A100 GPU with diffusion-based CogVideoX. While our model outperforms CogVideoX on VBench (see Table 1), it yields lower FLOPs and inference time for similar video sizes, demonstrating the superior inference efficiency of pyramid flow. Note that the gain in inference efficiency is essentially a by-product of the autoregressive designs to improve training efficiency.

  Model	Output video	FLOPs(G)	Speed(sec)
  CogVideoX-2B	720 x 480 x 49	47227	90
  CogVideoX-5B	720 x 480 x 49	169192	180
  Ours (384p 5s)	640 x 384 x 121	30154	62
  CogVideoX 1.5-5B	1360 x 768 x 81	-	1000
  Ours (768p 5s)	1280 x 768 x 121	112386	336

---

Reference:

[1] Yu, et al. Language model beats diffusion - Tokenizer is key to visual generation. ICLR 2024.

[2] Jin, et al. Video-LaVIT: Unified video-language pre-training with decoupled visual-motional tokenization. ICML 2024.

We sincerely appreciate the time and effort you have taken to review our paper and provide valuable feedback. Below are our responses to the concerns raised in your review.

[W1] Clarity of writing.

• Thanks for pointing that out. The "full resolution" refers to the vae latents used in traditional latent diffusion models (LDM), and the generation of our pyramid flow is done in latent space, not pixel space. We apologise for the misleading wording. We have revised the draft to clarify this and to change the grammatical errors or unnecessary terms, following your valuable suggestions.

[W2] Theoretical derivation.

• Thank you for the valuable suggestion. In the line before Equations (7) and (8), we mentioned "conditional probability path", which implies that the following two equations are conditioned on $x_1$. To avoid any ambiguity, we have revised the draft to state this explicitly. Note that this does not affect subsequent derivations, e.g. Equation (13), because it only involves taking the expectation on both sides of the equation.
• One thing to clarify is that the conditional flow within each window is actually conditioned on the real endpoint $x_1$ instead of the window endpoint $\hat{x}_{e_k}$. By this definition, the conditional distribution should be Gaussian.
• Compared to rectified flow, flow matching learns random couplings between the data $x_1$ and the noise $n$ (they are not the same distribution), which is less inference-efficient due to curved flow trajectories. However, rectified flow requires simulation using the flow ODE to derive optimal coupled training samples, which is much less training-efficient.

[W3-1] Benefit of coupling noise.

• The rationale for improving straightness by coupling noise is as follows: The straightness of the flow trajectory is usually compromised when there are intersections. Sampling the endpoints independently (as in vanilla flow matching) creates random directions for each trajectory and leads to intersections. Instead, by coupling the sampling of these endpoints, as in Equations (9) and (10), we can create more organised, possibly parallel trajectories with fewer intersections, thus improving straightness.
• We further illustrate this with a toy experiment in Figure 13 in the appendix, where coupling noise indeed leads to more straight flow trajectories.

[W3-2] Clarification of Equation (10).

• To clarify, Equation (10) is just an instantiation of Equation (8) that is used only in training. On the other hand, Equation (8) defines the entire flow model and is used in both training and inference. Therefore, the inference procedure in Section 3.2.2 and Appendix A is derived based on Equation (8) only.

[W3-3] Number of sampling steps.

• The number of sampling steps for each stage is set to 10. We have found that this setting achieves a good balance between total inference time and generation quality.

[W3-4] Temporally compressed pyramid flow.

• Temporally compressed pyramid flow should also work. It is not adopted in the paper because next-frame prediction is a more natural choice than next-scale prediction for video. The video frames are already well ordered along the time axis, and there is causality that facilitates autoregressive generation.

[W4] Clarification of video VAE.

• We did not evaluate the quantitative performance of the video VAE because it is not our main technical contribution, the whole architecture just follows MAGVIT-v2 [1]. We have compared our causal video VAE with other open source versions on $256 \times 256$ resolution 17-frame videos from the WebVid. From the results presented, we can see that our VAE achieves a comparable PSNR value to that of CogVideoX with a higher compression rate.

  Model	Compression	PSNR($\uparrow$)
  Open-Sora	$8 \times 8 \times 4$	28.5
  Open-Sora-Plan	$8 \times 8 \times 4$	27.6
  CogVideoX	$8 \times 8 \times 4$	29.1
  Ours	$8 \times 8 \times 8$	28.9

• We train a new video VAE primarily because the normalization design of open-source VAEs is not compatible with our pipeline. To natively support both T2V and I2V, the first frame of the video VAE latent should be identical to an image latent. A simple trick to ensure this is to normalize the first frame and subsequent frames separately, which has not been adopted in open source VAEs.