FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion

Response to Reviewer Sj6K

Thanks to the reviewer for the valuable comments.

Q1: Ablation study on input size and higher resolution

A1.1 : Yes, in our experiments, we setup the 81 frames consistent with official Wan Model.

A1.2: The quadratic complexity O(N²) of attention operations means that computational demands grow dramatically as we increase either the number of frames (temporal dimension) or the resolution (spatial dimensions). Our joint tile-wise FP8 quantization and structured sparsity are specifically designed to address this scaling challenge. As the attention computation becomes more expensive, the relative benefits of our optimizations become more pronounced.

To empirically validate this scaling behavior, we conducted comprehensive experiments across various video configurations:

These results clearly demonstrate our scaling properties:

Temporal Scaling: Fixing resolution at 480p, we observe progressively better speedups as video duration increases: 4.33× (41 frames) → 4.52× (81 frames) → 4.78× (121 frames). This improvement occurs because longer sequences have more temporal redundancy that our sparse attention patterns can reduce without quality loss.

Spatial Scaling: Fixing duration at 81 frames, speedups improve with resolution: 4.52× (480p) → 4.96× (720p) → 5.20× (1080p). Higher resolutions benefit more because our tile-wise approach maintains consistent hardware utilization while the baseline suffers from increased memory pressure and irregular access patterns.

In summary, FPSAttention's efficiency gains scale favorably with both temporal duration and spatial resolution.

Q2: Missing hyperparameter details (α₁, α₂, g_coarse, etc.)

A2: Our hyperparameter selection follows an empirical approach based on the error analysis shown in Figure 5. Specifically, we observe that error tolerance follows a characteristic U-shaped curve across denoising steps, which guides our three-phase partitioning. For the Wan models evaluated, we empirically determined α₁=0.3 and α₂=0.7, meaning early phase (0-30% of steps), mid phase (30-70%), and late phase (70-100%). The granularity and window sizes were selected through grid search on a held-out validation set, optimizing for the quality-speed Pareto frontier.

Regarding the concern about dataset sensitivity, our approach is designed to be robust across different generation scenarios. The key insight is that the error tolerance patterns (Figure 5) are inherent to the diffusion process itself rather than specific to particular datasets. The noise-dominated early steps and mid-steps exhibit consistent characteristics regardless of the content being generated. We validated this by testing our fixed hyperparameters across diverse VBench prompts without dataset-specific tuning.

The statement about "selected at inference time" (Line 227) may have caused confusion. To clarify: we perform one-shot hyperparameter selection during method development, then use these fixed values for both fine-tuning and inference. There is no need to fine-tune for different validation datasets - the same hyperparameters work across all our experiments.

Q3 : Hardware-optimized kernel contribution

A3: Our ablation study demonstrates that fused kernel implementation is crucial for achieving high efficiency. Without proper kernel optimization, theoretical FLOPs reduction poorly translates to actual speedup.

When FP8 quantization and sparsity lack native hardware support—using pseudo-quantization and tile-wise sparsity simulation—the system cannot fully exploit hardware capabilities. Our measurements show: FP8 quantization alone achieves 2.0× theoretical FLOPs reduction but only 1.4× actual speedup (70% efficiency); sparsity alone provides 3.5× theoretical reduction with 1.8× actual speedup (51% efficiency) due to irregular memory patterns.

The efficiency gap widens dramatically with naive combination. Despite 7.0× theoretical FLOPs reduction, naive implementation delivers merely 2.2× actual speedup—31% efficiency. This occurs because separate quantization and sparsity implementations introduce redundant memory operations, prevent data reuse, and fail to utilize specialized Tensor Cores. In contrast, our fused kernel achieves 4.96× actual speedup from the same 7.0× theoretical reduction (71% efficiency)—a 2.26× improvement over naive approach.

This improvement stems from hardware-conscious design that integrates quantization and sparsity within a single kernel. Through training-aware co-design, quantization and sparsity patterns align with hardware compute units. The fused kernel exploits this alignment by coalescing memory accesses, minimizing data movement across GPU hierarchies, utilizing FP8 Tensor Cores on Hopper architectures, and eliminating redundant computations.

The kernel design is indeed crucial for translating theoretical computational savings into practical speedups, and we provide a comprehensive ablation study to quantify its contribution in the table above.

Q4: Inconsistent reported model size

A4: We sincerely apologize the typo for this inconsistency. The correct model sizes are:

• Wan2.1-1.3B: 1.3 billion parameters, typically used for 480p generation
• Wan2.1-14B: 14 billion parameters (not 13B), typically used for both 480p and 720p generation

We will correct all instances of this error in the revised manuscript and ensure consistency throughout the paper and supplementary material.

Q5: Ablation study with different components

A5: Our design philosophy separates concerns between quality preservation (Denoising Step-Aware Scheduling) and speedup optimization (Hardware-Optimized Kernel Design), with each component serving a distinct and complementary purpose.

The denoising step-aware scheduling primarily preserves generation quality under aggressive compression. As shown in the table below, incorporating denoising step-aware scheduling with our core QAT+Sparse components improves VBench scores from 0.8019 to 0.8160 (+1.8% quality improvement) while maintaining comparable speed (2.18× vs 2.20×).

Meanwhile, the hardware-optimized kernel design translates theoretical computational savings into wall-clock speedups. As demonstrated in Section 3.4 and the table below, employing our hardware kernel dramatically accelerates inference from 2.20× to 4.96× while maintaining quality (VBench score 0.8046).

Thanks for helping us improve our manuscript. We will update these ablation studies in the revised version.

Response to Reviewer JYRy

Thanks to the reviewer for the valuable comments.

W1: Fine-tuning costs

WA1: First, our training cost is minimal compared to video generative model pretraining. As an analogy, while video generation models require massive pretraining resources—OpenAI's Sora is estimated to need 4,200-10,500 H100 GPUs for one month [1]. Our empirical statistics show that the training overhead accounts for less than 1% of the pre-training.

Second, training-free methods cannot achieve effective joint optimization of quantization and sparsity. As shown in Figure 2, naive combination leads to severe quality degradation because "quantization errors can be magnified when combined with sparsity mechanisms" (line 51-52). Our supplementary Table A quantifies this: training-free joint application causes a 21.1% performance drop (0.8019→0.6325), while FPSAttention achieves a 1.8% improvement.

As a byproduct, our training represents a one-time investment that is rapidly amortized through 4.96× end-to-end speedup, translating to around 80% reduction in recurring inference costs. For production models serving millions of requests, this upfront cost becomes negligible compared to the cumulative inference savings.

[1] Plappert, M. Under the hood: How OpenAI's Sora model works. Factorial Funds Blog.

Q1 : The proposed methods are engineering-oriented

A1: Our core contribution lies in a training-aware co-design of sparsity and quantization—a paradigm that is distinct from previous training-free approaches such as SageAttention and STA. We observe that naive combination of FP8 quantization and sparse attention leads to catastrophic 21.1% performance degradation (see Appendix B.4). Figure 2 in the paper also shows that training-free combination of FP8 quantization with STA produces visually corrupted outputs. To make FPSAttention effective, we innovatively propose three non-trivial strategies, including unifying tile-wise operations, denoising step-aware scheduling and a hardware-optimized kernel design. Regarding timestep-aware approaches like [1], we acknowledge this important prior work. Previous timestep-aware methods focus solely on quantization parameters, missing the critical interplay between precision and sparsity that varies across denoising phases. However, our denoising step-aware strategy fundamentally differs in that it adaptively schedules both quantization granularity g(t) and sparsity window size W(t) based on different compression error patterns across denoising steps.

Q2: Generalization to MMDiT

A2: Our FPSAttention targets the general attention module, thus it is intuitively applicable to common diffusion Transformer architectures, including MMDiT. Since the main difference between MMDiT architecture and Wan2.1 is the condition injection part, corss-attention v.s. in-context attention, the adaptation will be straightforward: our implementation already supports arbitrary token configurations, and the in-context attention, a pure self-attention module, simply requires adjusting our tile partitioning to accomodate modality boundaries while maintaining the same optimization principles. We anticipate that FPSAttention will deliver comparable or even superior speedups on MMDiT due to the longer sequence lengths from concatenated multimodal tokens, which amplify the benefits of our sparsity and quantization co-design. We will futher try our FPSAttention with models using MMDiT like HunyuanVideo [2].

[2] Kong, Weijie, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong et al. "Hunyuanvideo: A systematic framework for large video generative models." arXiv preprint arXiv:2412.03603 (2024).

Response to Reviewer jgQE

Thanks to the reviewer for the valuable comments.

Q1: Training and calibration costs compared to baseline sparse attention methods

A1: There is a fundamental difference between training-free sparse methods and our trainable joint optimization method FPSAttention. The methods in Table 2 are predominantly training-free approaches with zero additional training cost—SageAttention, SpargeAtten, and STA require no training, while Sparse VideoGen only needs minimal online profiling with 64 query tokens. Our FPSAttention is training-aware but delivers superior quality-efficiency trade-offs at a reasonable fine-tuning cost across two aspects:

First, our training cost is minimal compared to video generative model pretraining. As an analogy, while video generation models require massive pretraining resources—OpenAI's Sora is estimated to need 4,200-10,500 H100 GPUs for one month [1]. Our empirical statistics show that the training overhead accounts for less than 1% of the pre-training.

Second, training-free methods cannot achieve effective joint optimization of quantization and sparsity. As shown in Figure 2, naive combination leads to severe quality degradation because "quantization errors can be magnified when combined with sparsity mechanisms" (line 51-52). Our supplementary Table A quantifies this: training-free joint application causes a 21.1% performance drop (0.8019→0.6325), while FPSAttention achieves a 1.8% improvement.

As a byproduct, our training represents a one-time investment that is rapidly amortized through 4.96× end-to-end speedup, translating to around 80% reduction in recurring inference costs. For production models serving millions of requests, this upfront cost becomes negligible compared to the cumulative inference savings.

[1] Plappert, M. (n.d.). Under the hood: How OpenAI's Sora model works. Factorial Funds Blog.

Q2: Comparison with previous methods, including QVD and ViDiT-Q quantization

A2: We acknowledge that comparing with QVD and ViDiT-Q would strengthen our paper and commit to adding these comparisons in our revision.

The fundamental difference lies in our training-aware approach. QVD and ViDiT-Q employ post-training quantization (PTQ), while FPSAttention uses quantization-aware training (QAT). This eliminates the training-inference gap that limits PTQ methods. While QVD achieves near-lossless W8A8 quantization and ViDiT-Q reports 1.4-1.7× speedups, our training-aware joint optimization of quantization AND sparsity achieves 4.96× speedup by learning optimal patterns directly from data.

Our choice of FP8 over INT8 provides critical advantages for attention mechanisms. FP8 better captures the wide dynamic range of attention activations compared to INT8 used by QVD[2]/ViDiT-Q [3]. Moreover, FP8 operations are natively accelerated on modern GPUs (H100/H200), enabling our superior speedups [4]. While INT8 offers broader hardware compatibility, it can struggle with attention's dynamic range, particularly in regions with high sparsity [5].

We are conducting additional experiments comparing FPSAttention with ViDiT-Q. To best of our knowledge, the QVD is not opensource code yet, therefore we mainly focus on the experiment validation for ViDiT-Q. Preliminary results show that while both methods achieve comparable generation quality with timestep-aware strategies, our joint optimization delivers significantly higher speedups. This validates that quantization alone has inherent limits - by jointly optimizing with sparsity and leveraging hardware-native FP8, we push beyond these limits to achieve a superior speed-quality trade-off.

In our revision, we will add: (1) dedicated comparison experiments with ViDiT-Q, (2) analysis of how our denoising step-aware scheduling relates to ViDiT-Q's timestep-level quantization, and (3) discussion of these methods as complementary approaches serving different deployment scenarios.

Table: Comparison with ViDiT-Q on VBench Evaluation Metrics

Note: Values in parentheses show the change from respective baselines. Bold values indicate best performance.

Key Observations:

• ViDiT-Q shows minimal quality degradation (-0.20 to -1.39 on most metrics) while achieving 1.40× speedup
• FPSAttention demonstrates quality improvements on critical metrics (+3.88 Imaging Quality, +14.45 Dynamic Degree) despite aggressive compression
• Our method achieves 3.5× higher speedup (4.96× vs 1.40×) while maintaining or improving quality

This empirical evidence validates that joint training-aware optimization fundamentally improves the speed-quality trade-off, enabling simultaneous quality improvements and dramatic speedups that are impossible with quantization-only approaches.

[2] Tian, Shilong, et al. "QVD: Post-training Quantization for Video Diffusion Models." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.
[3] Zhao, Tianchen, et al. "Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation." arXiv preprint arXiv:2406.02540 (2024).
[4] Kuzmin, A., Van Baalen, M., Ren, Y., Nagel, M., Peters, J., & Blankevoort, T. (2022). FP8 Quantization: The Power of the Exponent. arXiv preprint arXiv:2208.09225.
[5] Noune, B., Jones, P., Justus, D., Masters, D., & Luschi, C. (2024). Efficient Post-training Quantization with FP8 Formats. arXiv preprint arXiv:2309.14592.

Q3: Impact of naive sequential combination of quantization and sparsity

A3: Direct combination of prior quantization techniques with sparse attention leads to catastrophic quality degradation. As shown in Figure 2, training-free combination of FP8 quantization with STA produces visually corrupted outputs, while our extensive ablation study (Supplementary Table A) reveals a devastating 21.1% performance drop (0.6325 vs baseline 0.8019), with critical metrics like Human Action collapsing to 0.02 and Multiple Objects to 0.0.

As explained in Lines 49-60 of the paper, "this failure may be caused by sparsity mechanisms that prioritize retaining tokens with high-magnitude attention scores, while quantization disproportionately introduces large errors in these values." This motivates our FPSAttention that jointly optimizes quantization and sparsity patterns through unified tile-wise operations in a training-aware manner. This co-design ensures quantization and sparsity work synergistically, enabling 4.96× speedup while maintaining visual quality - demonstrating that training-aware optimization is not merely beneficial but essential for aggressive compression in video diffusion models.

Q4: Limitations discussion

A4: We thank the reviewer for this suggestion. We would like to clarify that we have included a discussion of limitations in Section B "Limitation and Societal Impacts" in our Supplementary Material. Our limitations discussion addresses several key aspects: our approach works best with modern FP8-capable GPUs (e.g., NVIDIA Hopper architectures) though it can still provide benefits on older hardware; the method requires quantization-aware training which involves moderate additional training time compared to post-training approaches; the denoising step-aware scheduling includes several hyperparameters (α₁, α₂, quantization granularities, window sizes) that need optimization for different architectures; and while our current evaluation focuses on Wan2.1, the approach shows promise for extension to other video diffusion transformers. We will move the limitations discussion in the main paper of the revised version.

Dear Reviewer jgQE,

We are grateful for your thorough review and confirmation that our rebuttal have successfully addressed your concerns.

We deeply appreciate your continued strong support of our work, particularly your willingness to champion the paper. Your endorsement represents a significant vote of confidence in our research.

Thank you for your valuable insights and constructive engagement throughout the review process.

Sincerely,

Authors of Paper #6707

Response to Reviewer zqGz

Thanks to the reviewer for the valuable comments.

Q1 : Adaptability of FPSAttention to pre-trained models vs. training from scratch

A1: FPSAttention is specifically designed as a post-training adaptation method that fine-tunes already-trained models, not a training-from-scratch approach.

As stated in Section 4, we "implemented our proposed framework on the Wan2.1 architecture (1.3B and 14B variants), preserving the original model structure while introducing FPSAttention." The 7-day training period mentioned represents the fine-tuning phase where we jointly optimize FP8 quantization parameters and sparsity patterns starting from pre-trained checkpoints—this is significantly shorter than the months typically required for training such large models from scratch.

This post-training fine-tuning approach offers crucial advantages over both training-from-scratch and training-free methods. Unlike training-free approaches that suffer 21.1% performance degradation when naively combining quantization and sparsity (Table A in supplementary), our training-aware fine-tuning enables the model to adapt to joint quantization-sparsity constraints, achieving up to 4.96× end-to-end speedup while maintaining generation quality. The joint optimization during fine-tuning allows the model to learn optimal patterns that preserve critical information pathways despite aggressive compression.

Q2: Hardware compatibility and fallback strategies for non-FP8 platforms

A2: We acknowledge that FPSAttention's full performance benefits require modern FP8-capable hardware, but our joint optimization framework remains valuable across diverse platforms. For instance, on AMD MI300 series GPUs, we can perform deep custom development of the corresponding kernel, similar to the ROCm version of FlashAttention [2]. This demonstrates the potential of our methodology for application across different platforms. Looking forward, as newer architectures like NVIDIA's B200 series introduce FP6 capabilities, our training-aware co-design methodology can extend to these lower precisions, potentially achieving even greater speedups while validating the importance of hardware-native quantization design. For legacy hardware without native FP8 support (e.g., A100, V100), our unified optimization framework still provides substantial benefits through a fallback strategy: FP16/BF16 mode where our tile-wise sparsity patterns remain effective, achieving approximately 3× speedup from sparsity alone. Crucially, even without FP8 acceleration, our training-aware joint optimization prevents the catastrophic quality degradation seen in naive combinations (Table A).

Q3: FPSAttention performance with step-reduction and alternative sampling methods

A3: We thank the reviewer for this insightful question. It highlights a crucial distinction between two orthogonal axes of diffusion model acceleration: (1) optimizing the sampling schedule to reduce the number of steps (inter-step optimization), and (2) accelerating the computation within each denoising step (intra-step optimization).

Our work, FPSAttention, fundamentally belongs to the second category. It focuses on accelerating the core attention mechanism, which is a major computational bottleneck within every single forward pass of diffusion transformers. Methods like aggressive step-reduction (e.g., DeepCache) or efficient samplers, on the other hand, belong to the first category.

These two approaches are, by design, complementary rather than competitive. To empirically validate this principle, we integrated FPSAttention with DeepCache [3], a representative and state-of-the-art step-reduction method. The results are summarized below:

As shown in the table, DeepCache alone achieves a 2.3× speedup by caching and reusing features across adjacent steps. FPSAttention alone provides a 4.96× speedup by optimizing the expensive attention operations within each step. When combined, they yield a cumulative 8.02× speedup. Our method serves as a foundational enhancement to the attention module, which benefits any sampling process that relies on it. Besides, training-based step-reduction approaches like sCM [4], which require additional GPU hours for finetuning, are also theoretically orthogonal and can be further combined to achieve real-time generation. We believe this can serve as a future direction for the community.

Reference

[1] Plappert, M. (n.d.). Under the hood: How OpenAI's Sora model works. Factorial Funds Blog.

[2] ROCm/flash-attention

[3] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Deepcache: Accelerating diffusion models for free." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15762-15772. 2024.

[4] Lu, Cheng, and Yang Song. "Simplifying, Stabilizing and Scaling Continuous-time Consistency Models." The Thirteenth International Conference on Learning Representations.

Dear Reviewer zqGz,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,

Authors of #6707