Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile

We thank the reviewer for acknowledging the novelty of our attention map findings and the speedup of our approach. We summarize the questions here and respond to them individually:

Q1: No comparison with sequential/autoregressive video generation models that might be computationally cheaper.

A1: We thank the reviewer for discussing the literature on sequential/autoregressive video generation models. In addition to the mentioned literature, we also include more references in this line [1][2][3] and have updated the paper accordingly. A major line of these works simply invokes diffusion models in an auto-regressive loop. Our method accelerates a single diffusion forward pass, which is compatible with these methods. For instance, [1] and [2] can simply replace their original diffusion model with one that is tuned with our Efficient-vDiT method, and benefit from the speedup without needing to change the overall generation procedure.

[1] Henschel, R., et al. (2024). StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text. arXiv preprint arXiv:2403.14773.

[2] Xiang, J., et al. (2024). Pandora: Towards General World Model with Natural Language Actions and Video States. arXiv preprint arXiv:2406.09455.

[3] Zheng, Z., et al. (2024). Open-Sora: Democratizing Efficient Video Production for All. https://github.com/hpcaitech/Open-Sora

Q2: Lacks detailed ablation study showing separate effects of sparse attention without MLCD and experiment of first T_sparse then T_MLCD.

A2:

1. We updated the experiments with T_sparse first and then T_MLCD in Appendix D. We find that the T_sparse stage is orthogonal to the T_MLCD stage, observing only negligible differences in the VBench, and CD-FVD scores.

2. In Tables 6 and 7 (upper section), we present the evaluation results for distillation works applied independently to both the base model and MLCD, demonstrating the isolated effect of sparse attention. In Table 6, the CD-FVD scores for Base_{3:5} and Base_{4:4} consistently remain below 200. However, as the attention sparsity increases, the CD-FVD score rises to 322.28, indicating a degradation in video generation quality. A similar trend is observed in Table 7, confirming that sparse attention has comparable effects when applied to both the base model and MLCD model.

Q3: Both the sampling distillation (Stage 1) and knowledge distillation (Stage 3) lack technical novelty, as similar methodologies have been proposed in previous works [3,4]. The distinction between our proposed distillation and existing literature is unclear.

A3: Our main novelty is in the discovery of the analysis of attention tile redundancy patterns, where we adapted existing distillation processes to leverage this observation for inference acceleration.

Q4: Only shows two video examples with four frames each.

A4: We have included updated video examples from VBench and examples similar to those on OpenSora's website featuring dynamic scenes in Appendix E, demonstrating our capability to handle rapid, large-scale motions.

We thank the reviewer for acknowledging the novelty of our Attention Tile finding and layer-wise optimal search approach. We summarize the questions here and respond to them individually:

Q1: The main speedup comes from the existing MLCD method rather than the paper's novel contribution (sparse attention)

A1: We would like to respectfully clarify that our method achieves 2.83x speedup in the attention module (Table 1) and 1.77x end-to-end speedup (listed below), with a sparsity level of 1:7. Our key point here is that although the sparse attention alone does not achieve higher speedup than the MLCD method alone, this speedup is arguably substantial, and is complementary to MLCD. In 3D-DiT, redundancy stems from repetitive diffusion sampling and 3D attention; MLCD addresses the former, while attention distillation targets the latter. The value of our method is that it harmoniously connects them to effectively eliminate 3D-DiT redundancy.

Q2: Only experimented their method on one DiT based text-to-video generation model.

A2: We have updated ablation of the attention distill experiment on new model CogvideoX in new thread.

Q3: Lacks comparison with other acceleration methods, only showing self-comparisons with different parameter settings

A3: We thank the reviewer for the suggestion. We included one state-of-the-art work, PAB[1], into our discussion. PAB, developed on spatial-temporal DiTs, reuses attention computation from previous denoising steps to speed up inference. In their best-performing setup, they compute spatial attention, temporal attention, and cross attention every 2, 4, and 6 steps, respectively, which results in an average speedup of 4 in attention. This would translate to less than 4x speedup in an end-to-end setting. As shown in Table 5, our method, combined with consistency distillation, achieves 6.60x-7.80x end-to-end speedup. Second, we'd like to point out that PAB falls into the category of methods that leverage the repetitiveness between diffusion steps. Intuitively, it would be less compatible with consistency distillation (which is a well-established method to reduce repeated diffusion steps), because consistency distillation already reduces the repetitiveness between diffusion steps. In contrast, our proposed approach discovers and addresses a new type of redundancy in video diffusion, and shows ample evidence that it is complementary to consistency distillation.

[1] Zhao, X., et al. (2024). Real-Time Video Generation with Pyramid Attention Broadcast. arXiv preprint arXiv:2408.12588.

Q4: The diagonal attention pattern emphasized in the paper is an obvious phenomenon due to the basic property of self-attention

A4: While the high main diagonal values may be obvious to some audiences, this is only one part of the attention tile phenomenon. The attention tile phenomenon comprises four observations (Figure 1): repetitiveness, large diagonals, locality, and data independence. Such observations have not been revealed by other literature to the best of our knowledge, and form the basis of our methodology.

Q2: Only experimented their method on one DiT based text-to-video generation model.

A2: We appreciate the reviewer's feedback. We have added comprehensive experiments on the CogVideoX-5B model in Appendix D.2 to demonstrate our method's generalization capability. CogVideoX is based on the MM-DiT architecture, where its attention module concatenates text tokens with video tokens, which differs from Open-Sora-Plan's cross attention module. These experiments demonstrate that our method works effectively on both MM-DiT and Open-Sora-Plan's cross attention architectures.

1. Kernel Performance : We analyze the computation time for a single sparse attention kernel below. The results show that as sparsity increases, computation time decreases significantly. For instance, with a 2:11 attention mask, the execution time reduces to 15.16ms, achieving a 1.72× speedup compared to the full mask.

2. Evaluate our method on VBench

Experiment Setting: CogVideoX-5B is profiled using Algorithm 1. For training, the model is trained for 10,000 steps, equivalent to 10 epochs of the dataset. The learning rate is set to 1e-7, and the gradient accumulation step is 1. The diffusion scale factor $\lambda$ is set to 1.

Quantitative results: The VBench evaluation results of the knowledge distillation model are shown below. Our model's results are within 1% of the final score with no noticeable drop in several key dimensions. It achieves comparable performance to the original model.

Qualitative results: In Appendix D.2. Figure 7, we demonstrate that our method shows robust performance in processing dynamic, complex scenes while maintaining high-quality video output using the prompt from CogVideoX official website.

We thank the reviewer for the positive feedback on the sparse attention mechanisms in the video model and distillation framework. We summarize the questions here and respond to them individually:

Q1: Tradeoff analysis: Lacks detailed discussion of trade-offs between sparsity and video quality, especially for different types of content (static vs. dynamic scenes)

A1: From Table 5, we can observe that when sparsity increases, VBench dimensions such as total score and motion smoothness decrease, while higher sparsity leads to better approximation of the original image. Also, as shown in Table 1, as sparsity increases, the sparse kernel requires less execution time, thus reducing inference time. To address this trade-off, we present an optimization method based on the Lagrangian method in Appendix A, which considers how to achieve higher sparsity under given acceleration ratio constraints, thereby better approximating the original image. We have included updated video examples from VBench and examples similar to those on OpenSora's website featuring dynamic scenes in Appendix E.

Q2: Concerns about model generalization when using only 0.1% of pre-training data.

A2: We thankfully agree with the reviewer that more data would improve model generalization ability. In fact, we show that our method has already had good generalization ability with only 0.1% of pre-training data on VBench.Note that VBench already contains a diverse suite of test prompts, which suffices as a proof-of-concept. We believe distillation with more data will further boost the performance, but defer it as a future work due to the large amount of GPU hours required.

Q3: Current metrics (VBench and FVD) may not fully capture video quality, especially temporal coherence -> Content-Debiased FVD.

A3: Following your suggestion, we have updated our evaluation to include Content-Debiased FVD scores, which more effectively capture the temporal coherence of video quality. This additional metric provides better insight toward per-frame quality over temporal realism and helps identify its sources. As shown in the table below, the base model achieves better performance on Content-Debiased FVD, which aligns with our expectations.

Q4: Lack of sufficient video samples in supplementary materials.

A4: We have included updated video examples from VBench and examples similar to those on OpenSora's website featuring dynamic scenes in Appendix E, demonstrating our capability to handle rapid, large-scale motions.

We thank the reviewer for the positive feedback on the Attention Tile phenomenon analysis and distillation framework. We summarize the weaknesses and questions here and respond to them individually:

Q1: Attention distill improves little compared to MLCD

A1: Attention distillation is a method that reduces redundancy that MLCD cannot address and is orthogonal to it. The redundancy in 3D-DiT comes from two sources: the repetitive sampling of diffusion and the redundancy of 3D attention. MLCD can only address the first part, while the goal of attention distillation is to address the second part. In Tables 1 and 2, we show that attention distillation itself provides substantial speedup. In our method, these two components can be combined to jointly eliminate redundancy in 3D-DiT inference.

Q2: Dynamic Degree trend analysis

A2: As discussed in the VBench paper, models often exhibit trade-offs between temporal consistency and dynamic degree metrics. Quantitatively, we measure our models and variants across the overall VBench dimensions to justify the quality of our method. Qualitatively, we provide more samples in Appendix E from the VBench prompt to demonstrate that our method improves motion dynamics while maintaining acceptable consistency. Overall, with higher sparsity, our method captures richer motions between frames but with a trade-off in temporal consistency.

Q3: Discrepancy between Table 1 and Table 7 results

A3: The difference between Tables 1 and 7 comes from different measurement scopes. Table 1 measures only the attention operation speedup, while Table 7 shows the speedup of the entire model (including overhead from normalization and MLP layers). For example, with the 1:7 ratio, while attention alone achieves a 2.83x speedup, the full model only achieves a 1.77x speedup due to other components.