Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

We sincerely appreciate your insightful feedback and the opportunity to discuss our work further. Your comments have been invaluable in refining our approach and clarifications.

Weakness 1: In-depth technical analysis and underlying mathematical principles or theoretical insights that govern these sparse patterns in the context of video diffusion models.

Answer: Here we provide the theoretical analyses of SVG. Consider a video with N frames, each containing F tokens. Every token can be encoded by an indices pair $(i, j)$, where $0 \leq i < N, 0 \leq j < F$. In the attention map we will flatten these two-dimensional indices into a 1D vector. Corresponding formula is $x = i \cdot F + j$.

Spatial Head:

For spatial head, let $a_1 > 0$ denote the threshold for spatial closeness between tokens. The spatial attention mask is defined as:

$f_{s}\left( (i_1, j_1), (i_2, j_2) \right) = 1 \text{ if } |i_1 - i_2| < a_1, \text{ otherwise } 0$

For flattened indices $x_1 = i_1 \cdot F + j_1$ and $x_2 = i_2 \cdot F + j_2$,
$f_{s}(x_1, x_2) = 1$ is equivalent to: $| \lfloor \frac{x_1}{F} \rfloor - \lfloor \frac{x_2}{F} \rfloor | < a_1$

The resulting attention map takes the form of block-banded structures: the attention mask will be on the main diagonal and also on neighboring $\pm(a_1 - 1)$ diagonals.

Temporal Head

For temporal head, let $a_2 > 0$ denote the threshold for temporal closeness between tokens. The temporal attention mask is defined as:

For flattened indices $x_1 = i_1 \cdot F + j_1$ and $x_2 = i_2 \cdot F + j_2$,
$f_{t}(x_1, x_2) = 1 \Leftrightarrow | (x_1 \bmod F) - (x_2 \bmod F) | < a_2 \Leftrightarrow | x_1 - F \cdot \lfloor \frac{x_1}{F} \rfloor - \left(x_2 - F \cdot \lfloor \frac{x_2}{F} \right) | < a_2 \Leftrightarrow \exists k, 0 <= k < 2N - 1, |(x_1 - x_2) - k F| < a_2$

The resulting attention map forms $2N - 1$ slanted diagonals that align along constant column index differences. These diagonals, often referred to as “slashes,” correspond to the token positions sharing similar temporal locations across frames. The corresponding points in the attention matrix yield a “slash-wise” pattern with width $a_2$.

In addition, we would like to emphasize that our work also provides other technical depth beyond the proposed online profiling and layout transformation. Our contributions comprise four main aspects.

(1) We identify the spatial and temporal sparse patterns, which unveil the potential for efficient acceleration.

(2) We propose algorithmic innovations, including online profiling strategy and layout transformation that exploit these patterns.

(3) We co-design highly optimized kernels with our algorithm to translate theoretical computation saving to on-hardware speedup.

(4) We present thorough evaluations that SVG can achieve an end-to-end speedup of up to 2.33x while maintaining high visual quality with a PSNR of up to 29.

Weakness 2: Comprehensive exploration of why these patterns emerge at a fundamental level.

Answer: In essence, these patterns arise because the pixels in a video often exhibit high similarity both within a single frame (spatially) and across consecutive frames (temporally). As similar patches in a video share nearly identical embeddings in the Q/K/V space, the self-attention mechanism naturally assigns high attention scores to these similar patches.

Spatial Head: Within a single frame, neighboring pixels (and patches) often share similar colors, brightness values, and gradual transitions. If a patch in one region of a frame has a high attention score to itself, it tends to exhibit high attention to other patches in the same frame that share similar visual patterns.

Temporal Head: Consecutive frames in a video, especially in static or slowly changing scenes, exhibit high similarity for the same patch location across time. If a particular patch has a strong correlation with itself (i.e., high attention to its own embedding), it will also place high attention on patches in other frames that share similar visual features.

Traditional video compression techniques have similarly leveraged spatial and temporal redundancy to reduce data size and complexity. For example, H264 (used in MP4) uses intra-prediction and discrete cosine transform (DCT) to reduce spatial redundancy and uses inter-prediction and recording only the difference (motion vectors and residuals) between consecutive frames to reduce temporal redundancy. These well-known strategies highlight that video data inherently contains a high degree of spatial and temporal redundancy—precisely the property our sparse videogen approach leverages.

Weakness 3: No video sample.

Video samples showcasing the results of our method can be accessed at the following anonymous link: https://drive.google.com/drive/folders/1jhEpZ69bKfyZWmoy63iS3FhECNnX-AZU?usp=sharing

We thank the reviewer for acknowledging our contributions and the insightful comments. We respond to the questions below. We will revise the paper's final versions based on the reviewers' comments.

How does the performance / efficiency compare to models interleaving spatial attention and temporal attention (2D + 1D Attention)?

Answer: Recent state-of-the-art models [1,2,3] have shifted toward using 3D full attention, which significantly improves performance at the cost of higher computational complexity. In contrast, although interleaved spatial and temporal attention [4,5,6] is computationally efficient, it suffers from limited modeling capability—especially in capturing complex spatiotemporal dependencies—leading to lower generation quality.

Our method delivers video generation quality on par with full 3D attention models while operating at a significantly lower computational cost. Compared to 2D + 1D approaches, SVG achieves not only substantial improvements in visual fidelity but also stronger temporal consistency. In terms of efficiency, SVG sits between 2D + 1D and full 3D attention—achieving a compelling balance of speed and quality. For example, as shown in the table below, OpenSora-v1.2—a representative open-sourced model utilizing 2D+1D attention—achieves significantly lower VBench scores than Wan2.1, a 3D model of similar parameter size. By reducing attention redundancy, SVG effectively decreases attention FLOPs by 2.55x while preserving superior generation quality. Thus, SVG presents a practical and efficient alternative for high-quality video generation.

[1] Kong, Weijie, et al. "Hunyuanvideo: A systematic framework for large video generative models." arXiv preprint arXiv:2412.03603 (2024).

[2] Wang, Ang, et al. "Wan: Open and Advanced Large-Scale Video Generative Models." arXiv preprint arXiv:2503.20314 (2025).

[3] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

[4] Zheng, Zangwei, et al. "Open-sora: Democratizing efficient video production for all." arXiv preprint arXiv:2412.20404 (2024).

[5] Lin, Bin, et al. "Open-sora plan: Open-source large video generation model." arXiv preprint arXiv:2412.00131 (2024).

[6] Ma, Xin, et al. "Latte: Latent diffusion transformer for video generation." arXiv preprint arXiv:2401.03048 (2024).

Thank you for your valuable suggestions and questions. Below, we address each point raised.

Q1: Can the attention layers in multi-view diffusion models also be accelerated?

Answer: Recent multi-view diffusion models such as MVDream [1] and CAT3D [2] have introduced 3D attention mechanisms to improve cross-view consistency and spatial coherence. MVDream takes an initial step in this direction but is limited by a relatively short context length (≤ 4k tokens), which diminishes the practical benefits of attention acceleration. CAT3D adopts finer-grained image representations and extends the attention span in 3D. It goes further than MVDream by applying 3D attention over longer sequences and finer feature maps. CAT3D also highlights the critical scalability issue: applying full 3D attention to high-resolution feature maps (e.g., 64×64) leads to prohibitively large sequence lengths—up to 32k tokens—making the approach computationally expensive. As a result, CAT3D restricts full 3D attention to lower-resolution feature maps (32×32 and below), where the overhead remains manageable.

This design underscores the computational bottlenecks present in naive 3D attention and presents a clear opportunity for approaches like SVG. Since SVG can leverage the spatial and temporal redundancy, it could potentially enable efficient 3D full attention at larger resolutions, unlocking performance and scalability gains that current methods avoid due to cost. However, CAT3D does not open-source their weights. We are willing to validate the effectiveness of SVG on them when the weights are available.

[1] Shi, Yichun, et al. "Mvdream: Multi-view diffusion for 3d generation." arXiv preprint arXiv:2308.16512 (2023).

[2] Gao, Ruiqi, et al. "Cat3d: Create anything in 3d with multi-view diffusion models." arXiv preprint arXiv:2405.10314 (2024).

Thanks for your valuable comments and feedback. We respond to the questions below.

Experimental Designs Or Analyses 1: The submission currently does not provide actual video demos.

Answer: As requested by the reviewer, we include actual video demonstrations at the following anonymous link: https://drive.google.com/drive/folders/1jhEpZ69bKfyZWmoy63iS3FhECNnX-AZU?usp=sharing. Our method achieves nearly lossless video quality.

Experimental Designs Or Analyses 2: How does the method perform in more challenging scenarios? Such as those involving complex spatial-temporal dynamics.

Answer: As suggested by the reviewer, we additionally include some representative video results of complex spatial-temporal dynamics in the following anonymous link: https://drive.google.com/drive/folders/1xscj4RcaOE-PeAXq6yiiUFqnTJqS5kaV?usp=sharing. Our method maintains high video fidelity (PSNR=28.624) under these challenging cases, such as scenes involving rapid motion and dynamic multi-person interactions.

Weakness 1: Insufficient clarity regarding the implementation specifics of the proposed online profiling strategy. When this profiling occurs, and does the profiling dynamically vary across different prompts, layers, or diffusion timesteps?

Answer: Thank you for the valuable feedback. In our implementation, online profiling is performed at every diffusion step and every layer right before the 3D full-attention computation. This ensures that SVG dynamically adapts to the different prompts and spatial-temporal patterns at each step.

We would like to emphasize that dynamic online profiling is essential to our method. We find that attention patterns exhibit substantial variation, especially across different steps and prompts. Static profiling patterns—whether fixed across layers, steps, or prompts—will lead to performance degradation, as demonstrated in Table 1, measured by PSNR, SSIM and LPIPS. The results are averaged over 10 examples. Our SVG uses the dynamic profiling strategy, consistently outperforming all static configurations.

Table 1

Static profiling patterns lead to significant quality degradation due to inaccurate identification of attention patterns. To illustrate this, we analyze the variation in attention patterns across different layers, diffusion steps, and data examples (prompts) by computing cosine similarity. Specifically, we employ oracle profiling to identify attention patterns for each head across different configurations. For example, we calculate the variation across different layers by computing a tensor with shape [num_layers, num_heads]. Each element in this tensor is labeled either -1 (indicating a spatial head) or 1 (indicating a temporal head). By calculating the cosine similarity between columns of this tensor, we quantify the variation of attention patterns. As demonstrated in Table 2, attention patterns exhibit substantial variation, necessitating a dynamic online profiling method.

Table 2