Fast Video Generation with Sliding Tile Attention

We sincerely thank Reviewer BHCc for their supportive feedback and genuine interest in our work.

Will the code be released to ensure reproducibility

We appreciate your emphasis on reproducibility. We confirm that all the code, configurations, and scripts required to reproduce our experiments will be fully open-sourced. This will ensure the community can easily replicate and build upon our results.

Will the proposed method have the potential to be applied to 3D or 4D generation?

We appreciate your insightful question regarding STA's potential applicability beyond standard video generation. Videos are already a type of 3D data (time × height × width). We believe STA naturally generalizes to other forms of 3D data such as 3D object generation (e.g., voxel grids) or even 4D data generation (e.g., dynamic point clouds or time-evolving 3D structures), so long as those data conform to some degree of data locality.

We thank the reviewer nmHo for their constructive feedback. Below, we address specific comments one by one.

More training details about STA w. Training are missing.

We have provided the primary details of STA training, including datasets, prompts, learning rate, and hardware specifications, in Appendix Section B and briefly mentioned them in line 265 of the manuscript. If there are specific additional details you would like us to include beyond these, we would be happy to provide them. We will also open-source all of our training code, configurations, and scripts to facilitate reproducibility.

No demo videos are provided in the supplementary files

Thanks for raise this concern. We provide an anonymous link to 20 sampled videos from our human evaluation experiments here: https://drive.google.com/drive/u/4/folders/1kRDt4ahiYQj1zk593FE6Hg2CgNsJcdIf

For video generation, only HunyuanVideo is used to conduct experiments.

Following your valuable suggestion, we further validated STA on Wan2.1 in a training-free setup, as shown in the results below. Together with FLUX's result in Table 5 of our manuscript, we believe they provide broader evidence of STA's generalizability.

Resolution: 93X1024X1024

Resolution: 93X768X1280

The results confirm STA's applicability across different model architectures.

We truly appreciate your thoughtful review. If our response has addressed your concerns, we would be grateful if you might consider a higher score. If there are any remaining points we could further clarify or improve, we would be sincerely thankful for your guidance.

We sincerely thank Reviewer C6tW for their insightful suggestions and valuable questions. Below we address your comments and strengthen our paper.

Are the results generalizable to other models than Huanyuan-Video?

As suggested, we further validated STA's generalizability beyond Huanyuan-Video. In our original manuscript, we demonstrated the effectiveness of STA on image-generation models like FLUX. Here, we additionally apply STA to Wan 2.1 in a training-free setup.

Resolution: 93X1024X1024

Resolution: 93X768X1280

The above results confirm the generalizability of STA beyond Huanyuan-Video. On Wan 2.1, STA achieves consistent speed-ups across different step settings and resolutions without compromising perceptual quality.

Are the optimization hardware dependent?

STA itself is not hardware dependent, and we designed our method to be easily deployable across GPU architectures:

A100/RTX 4090

Using the FlexAttention backend, STA achieves substantial speedups with minimal overhead, as demonstrated in the table below:

RTX 4090:

A100:

On H100 GPUs

We further leverage Tensor Memory Accelerator (TMA) capabilities to achieve even better performance. Specifically, STA implemented in TK on H100 achieves a 1.43× improvement compared to FlexAttention implementation, showcasing additional optimization potential with modern hardware.

In summary, STA does not inherently depend on specific GPU architectures. While advanced hardware features like TMA can further boost STA's performance, using FlexAttention alone provides straightforward and efficient speedups across other GPUs.

Extend the current relative work section with more literatures on attention based speed-up that are more related to the paper, instead of other types of speed-up.

We appreciate your suggestion to enhance the related work section of our manuscript. Should the paper be accepted and additional space permit, we plan to incorporate discussions on the following areas:​ 1. Efficient Vision Transformers (ViTs) with a focus on attention mechanisms such as Swin Transformer and EfficientViT. 2. Efficient attention mechanisms in large language models such as StreamingLLM and H2O.​ 3. Attention quantization techniques such as SageAttention.​

We appreciate Reviewer 3sP9's insightful question regarding the constraints on input resolutions and aspect ratios for Sliding Tile Attention (STA). Below, we clarify and expand upon these points:

I think STA should have restrictions on input resolution, length, and aspect ratio. The authors should clarify this key issue.

Indeed, STA requires input video latents to have dimensions that are integer multiples of the tile size to fully achieve the promised efficiency gains from sparsity. We briefly mentioned this constraint in line 214 of our submission. Specifically, in our current implementation, we use a tile size of (6, 8, 8), meaning the optimal latent dimensions are multiples of these numbers, i.e., (a × 6, b × 8, c × 8) where a, b, and c are positive integers.

For video latents whose dimensions do not strictly satisfy this requirement, we suggest two practical approaches to address this limitation:

Padding with Masking:

When generating a video with a dimension slightly different from multiples of the tile size, we can introduce padding tokens to match the tile-aligned dimension (which is a standard practice that is also used in FlashAttention kernel when the input sequence length is not an integer multiple of block size.) For instance, to generate a video with dimensions (29, 45, 76), we would pad the latents to size (30, 48, 80). In practice, padding tokens are then masked during attention computation to avoid contaminating the attention.

Cropping:

Alternatively, one can generate a slightly larger latent (e.g.,(30, 48, 80)) and subsequently crop the result to the desired dimensions.

Although both methods introduce computational overhead due to processing padding tokens or larger dimensions, this cost remains minimal compared to STA’s efficiency gains. To thoroughly address your question, we conducted additional experiments under multiple scenarios and resolutions. Below, we show the kernel benchmark results of applying STA to a (29, 45, 76) latent on RTX 4090 by padding to (30, 48, 80):

Our evaluations suggest this padding overhead is negligible compared to the substantial latency reduction achieved by STA. We also present end-to-end results applying STA to Wan 2.1 at a different resolution (93 frames × 1024 × 1024) in a training-free setup, further demonstrating STA is applicable to varying input dimensions:

We will explicitly clarify these practical constraints and solutions in the revised manuscript to better inform readers. We hope the additional experimental results address your concern and clarify the constraints involved. If our response resolves your reservations, we would be grateful if you would consider raising your score. We also welcome any further questions or suggestions you may have.