Radial Attention: $\mathcal O(n \log n)$ Sparse Attention for Long Video Generation

Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.

How is Figure 3 Obtained?

The attention scores shown in Figure 3 are averaged across all attention heads, layers, and timesteps, based on 10 diverse prompts from VBench [1]. These prompts cover a range of themes, including animals, sports, people, et cetera.

Does Fig 3. also Hold for Wan2.1 14B or Open-Sora?

Yes, both the attention decay phenomenon and our proposed exponential decay model hold for Wan2.1 14B [2] and Open-Sora [3], demonstrating that proposed "Spatiotemporal Decay" is universal for leading video generation models.

For Wan2.1 14B [2], we conducted a quantitative regression analysis on the average attention using 10 diverse prompts from VBench [1]. Specifically, we fitted three functions to the distribution:

• the inverse model $y = \frac{a}{(x-b)} + c$,
• the inverse square model $y = \frac{a}{(x-b)^2} + c$
• our proposed exponential decay model $y = \exp(-ax + b) + c$.

The regression results, measured by $R^2$ (higher is better), is summarized in the table below. Our exponential decay model achieves the best fit for both spatial and temporal decay, with $R^2$ values up to 0.996.

We applied the same analysis to Open-Sora 2.0, and again our exponential model achieved the highest $R^2$ values—over 0.992 in both decays.

These results demonstrate that both models follow the exponential Spatialtemporal Energy Decay.

Robustness of Spatiotemporal Energy Decay in Complex Scenarios

As requested by the reviewer, we validate our exponential decay model in the complex scenarios. We collected 10 high-motion prompts (e.g. dancing, competitive sports, and culinary action) with detailed and complex descriptions. Here we provide an example prompt.

breakdancer caught mid-freeze, balancing on one hand while the crowd forms a vibrant, cheering circle, energy radiating through low ambient street lighting.

Similarly, we average all the attention maps of Wan2.1 14B [2] on these prompts and then perform the regression analysis on the attention scores. The results are reported below. Our exponential decay model remains the best fit, achieving up to 0.997 $R^2$, demonstrating that the decay pattern holds in complex scenarios.

We also evaluated the visual quality of Aura Attention on these complex prompts for default-length generation. As shown in below table, the similarity metrics demonstrate that our approach maintains high video fidelity on both VBench prompts used in our paper and the complex prompts, indicating robustness of our method.

We also compare Aura Attention against the full attention for 4$\times$ longer video generation of HunyuanVideo after LoRA tuning. It is 4.4× faster in training and 3.7× faster in inference, while also delivering a slightly higher Vision Reward score:

Typos

Thanks for pointing these out! We will correct them in our manuscript.

[1] VBench: Comprehensive Benchmark Suite for Video Generative Models

[2] Wan: Open and Advanced Large-Scale Video Generative Models

[3] Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k

Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.

Complexity with Respect to Spatial Resolution

While Aura Attention has a theoretical complexity of $\mathcal{O}(n^2)$ with respect to spatial resolution, its practical speedup does not degrade as resolution increases for a fixed video length. As discussed in Section 4.2, our mask design guarantees constant sparsity across resolutions for a fixed number of frames. The table below compares the per-step latency of Wan2.1 14B using original full attention versus our Aura Attention for generating 5-second videos. The speedup becomes more significant at higher resolutions, since attention computation scales quadratically with the number of pixels per frame, while most other operations scale linearly.

Moreover, recent advances in video VAE have achieved remarkable progress in spatial compression, pushing the compression ratio from $8\times8$ to $16\times16$ [2] or even $32\times32$ [3]. These methods can be combined with Aura Attention to mitigate the quadratic cost of spatial resolution.

Short Video Generation

We would like to clarify that in Table 1 Aura Attention achieves video quality comparable to SVG [1] and outperforms STA [4] on Hunyuan Video under the same computation budget. On Wan2.1 14B, Aura Attention beats both baselines in quality. Regarding latency, Aura Attention matches SVG on both models. STA uses FlashAttention-3 (FA3) [5] as its backend, providing slightly higher speedup. Aura Attention can reach similar performance by switching the backend from FA2 to FA3, achieving a speedup of 2.27× on Hunyuan Video and 1.98× on Wan2.1 as shown in the below table, on par with STA.

Ablation Studies

As requested by the reviewer, we provide additional ablation studies to justify our design choices. The experiments were conducted on Wan2.1 14B at a resolution of $1280\times768$ with 117 frames. We will include them in our manuscript.

Attention Sink

We follow SVG [1] to preserve attention to the first frame (attention sink), which retains a significant portion of the attention energy. Quality metrics listed in the table below show that removing the attention sink could result in noticeable degradation of video quality. Here we align the mask sparsity with and without the attention sink for fair comparison.

Width of Central Band

The table below shows results of different central band widths. A central band width of $d$ means attention between frames $i$ and $j$ where $|i - j| \leq d$. For fair comparison, we align mask sparsity across all settings. Narrower bands (e.g., $d=0$) are suboptimal as they miss some important interactions between neighboring frames, which is crucial for video quality, while wider bands (e.g., $d=2$) spend too much computation budget on the central areas, failing to capture the critical tokens in the distant frames.

Width Scale Between Consecutive Bands

The table below shows results of different width width scale between consecutive bands. For fair comparison, we also keep mask sparsity consistent across all settings. Our width scale of 2 achieves the best video quality. Smaller scales (e.g., 1.5) reduce attention on the central band, while larger scales (e.g., 3 or 4) limit computation on important tokens in distant frames, making them suboptimal.

Whether to Reduce the Diagonal Frequency when the Width is below 1

We reduce the diagonal frequency when the width is below 1 to maintain $\mathcal{O}(n\log n)$ complexity. Otherwise, long videos would incur quadratic computation cost with respect to the number of frames, despite potential quality gains. Therefore, this ablation is not feasible in our design choice. We appreciate your understanding and are open to further discussion on this.

Why Aura Attention is Better for Longer Video Generation

For long video generation, the large number of tokens makes dense attention inefficient and hard to focus on key interactions. Aura Attention addresses this by leveraging the Spatiotemporal Energy Decay to preserve only important interactions, improving both learning efficiency and performance.

[1] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025

[2] Wan 2.2: Leading AI Video Generation Model

[3] Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k

[4] Fast Video Generation with Sliding Tile Attention, ICML 2025

[5] FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.

Complexity with Respect to Spatial Resolution

While Aura Attention has a theoretical complexity of $\mathcal{O}(n^2)$ with respect to spatial resolution, its practical speedup does not degrade as resolution increases for a fixed video length. As discussed in Section 4.2, our mask design guarantees constant sparsity across resolutions for a fixed number of frames. The table below compares the per-step latency of Wan2.1 14B using original full attention versus our Aura Attention for generating 5-second videos. The speedup becomes more significant at higher resolutions, since attention computation scales quadratically with the number of pixels per frame, while most other operations scale linearly.

Moreover, recent advances in video VAE have achieved remarkable progress in spatial compression, pushing the compression ratio from $8\times8$ to $16\times16$ [2] or even $32\times32$ [3]. These methods can be combined with Aura Attention to mitigate the quadratic cost of spatial resolution.

Robustness to Complex Spatiotemporal Dependencies

To evaluate whether our exponential decay model can capture complex spatiotemporal dependencies, we sampled 10 prompts from the LongVILA [4] supervised fine-tuning dataset. These prompts describe complex real-world scenes that demand high spatiotemporal dependencies due to intricate interactions and dynamic changes over time. A sample prompt is displayed below.

The woman shows the stickers to the camera, talks about them, places them on the table, and at one point sticks a sticker on her hand.

We then conducted a quantitative regression analysis on the average attention score distribution of Wan2.1 14B on these prompts. Specifically, we fit three models to the spatial and temporal attention decay curves:

• inverse model $y = \frac{a}{(x-b)} + c$
• inverse square model $y = \frac{a}{(x-b)^2} + c$
• our proposed exponential decay model $y = \exp(-ax + b) + c$.

The regression results, measured by $R^2$ (higher is better), is summarized in the table below. The attention score distribution nicely fit our exponential decay model, achiving $R^2$ up to 0.992. This means even for complex real-world scenes, our model still holds. See the following response for the video quality.

Capability of Capturing Long-Range Dependencies

As mentioned in Section 5.1 in our paper, we preserve full attention in the first two layers during both training and inference, and apply a 2-step full-attention warmup during inference. This setup helps the model effectively capture global information. Additionally, stacking multiple Aura Attention layers expands the receptive field, allowing information to propagate across distant tokens. Therefore, our method is capable of capturing long-range dependencies even in dynamic scenes.

To support this, we further evaluate default-length video generation on Wan2.1 14B using the aforementioned complex prompts from LongVILA [4]. As shown in below table, the similarity metrics demonstrate that our approach maintains high video fidelity on both VBench prompts used in our paper and the LongVILA prompts, indicating robust performance across different scenarios.

We also report the video quality of 4$\times$ longer video generation on HunyuanVideo using LongVILA prompts in the below table. Aura Attention matches the Vision Reward of full attention, demonstrating strong long-range modeling capabilities.

Additional Linear Attention Baselines

As suggested by the reviewer, we adapted two more image-based linear attention methods—LinFusion [5] and Focused Linear Attention [6]—for video generation. The table below shows their performance on 4× longer video generation after LoRA fine-tuning on HunyuanVideo, measured by Vision Reward. As mentioned in our paper, due to the substantial architectual discrepency between the linear attention and the original softmax attention, these baselines fail to the learn the weight distributions during the fine-tuning.

[1] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025

[2] Wan 2.2: Leading AI Video Generation Model

[3] Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k

[4] LongVILA: Scaling Long-Context Visual Language Models for Long Videos, ICLR 2025

[5] LinFusion: 1 GPU, 1 Minute, 16K Image

[6] FLatten Transformer: Vision Transformer using Focused Linear Attention, ICCV 2023

Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.

Quantitative Analysis on "Spatiotemporal Energy Decay"

To validate the exponential decay modeling of our proposed "Spatiotemporal Energy Decay" phenomenon, we conducted a quantitative regression analysis on the average attention score distribution of Wan2.1 14B using 10 diverse prompts from VBench [1]. Specifically, we fit three functions to the distribution:

• the inverse model $y = \frac{a}{(x-b)} + c$,
• the inverse square model $y = \frac{a}{(x-b)^2} + c$,
• our proposed exponential decay model $y = \exp(-ax + b) + c$.

The regression performance, measured by $R^2$ (higher is better), is summarized in the table below. Our exponential model achieves the highest $R^2$ values in both spatial decay and temporal decay, up to 0.996. This indicates that our proposed decay modeling is an excellent fit to the data.

Ablations on Initial Full-Attention Layers and Steps

As requested by the reviewer, we provide additional ablation studies to justify our design choices. We will include them in our manuscript.

For default-length video generation, we follow SVG [2] to apply full attention to the first 25% of timesteps (12 steps) as a warmup for all methods. We ablate this setting in the table below. For fair comparison, we match the overall computation of all settings by adjusting the sparsity of our Aura Attention mask. The 12-step warmup achieves the best video quality across all metrics.

For 4× longer video generation, we apply full attention to the first 2 steps as a warmup during inference. The table below shows the impact of different warmup steps on HunyuanVideo. Similarly, we match computation of all configurations. Using 2 warmup steps achieves the highest Video Reward.

To better capture global information, we keep the first two layers as full attention during training. The table below reports results on HunyuanVideo when using 0, 1, 2, or 3 dense layers, under the same computation budget. Our choice of using 2 full-attention layers delivers the best video quality.

[1] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024

[2] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025