$\text{S}^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation

We thank you for your positive feedback and comments. We respond to the concerns below:

Q1: Writing improvement about Fig.5, "FP", and Eq.(4) to Eq.(6).

A1: Thank you for your valuable suggestions. We will modify our statements according to your requirements in the revised version to make them clearer and more readable.

Q2: Computational overhead about the Hessian approximation.

A2: We report the computational burden of Hessian approximation across different scale video generation models. In fact, the Hessian approximation is very efficient, and will only bring minor computational burden ($\le$ 0.1%).

We reported the time increase caused by using the Hessian approximation when constructing the calibration dataset in the following table:

It can be seen that the computational burden of using Hessian approximation is minor, but it can bring significant performance improvement. We use the Levenberg-Marquardt approximation [1][2] to calculate the Hessian approximation, which requires only one step matrix multiplication to obtain the approximate result, and is very efficient.

Reference:

[1]. Frantar, et al. Optimal brain compression: A framework for accurate post-training quantization and pruning. NeurIPS 2022.

[2]. Frantar, et al. Gptq: Accurate post-training quantization for generative pre-trained transformers. ICLR 2023.

Q3: Computational overhead about the attention scores calculation.

A3: We report the computational burden of the attention scores calculation across different scale video generation models. The attention scores calculation is very efficient, and will only bring minor computational burden ($\le$ 0.7%).

We reported the time increase caused by calculating the attention scores during calibration process in the following table:

During the calibration process, we only need to use the Full-Precision model to conduct a single forward calculation of attention scores for all data in advance. When optimizing the quantization model, we can directly get the pre-computed attention scores by the data index, which brings minimal burden.

Q4: In the W4A6 setting, why are activations quantized to 6 bits while weights use only 4 bits? Some rationale or empirical justification would help clarify this design choice.

A4: Since our main baseline method ViDiT-Q [1] has explored the high bit quantization of the video generation model (e.g., W8A8 and W6A6) and almost achieved the lossless performance, continuing to experiment under high bit cannot fully reflect the differentiation. Also, lower bits can bring greater model compression effect and fewer bit operations, which is more significant for the model compression and acceleration. Therefore, we target the 4-bit weighted quantization that can bring higher compression ratio but is still heavily underexplored.

We first explored W4A4, a full 4-bit quantization setting. This is a very radical setting for Post-training Quantization, but our method still made great improvement compared with existing methods (See Tab. 2, Line 182) with only minimal performance drop (~1%).

We then hope to guarantee the lossless model performance under 4-bit weight quantization so that it can be widely used in practice. We found that under W4A6 setting, the existing quantization methods still show significant performance degradation, but our method can guarantee almost lossless performance. Therefore, we further conduct W4A6 quantization experiments in Tab.1 (See Line 164). We further reduce the quantization bit of the lossless video generation model to W4A6 and set a new benchmark for video model quantization.

Reference:

[1]. Zhao, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. ICLR 2025.

Q5: Difference between cross-attention and self-attention mechanisms.

A5: The experiment of CogvideoX and HunyuanVideo in this paper use a single 3D-attention on the architecture, so there is no need to distinguish attention mechanisms. For models that distinguish attention mechanisms, such as Wan2.1 [1], we only use the results of self-attention. We give the detailed explanation in the following paragraph. And we will add the explanation of this part in the revised version.

At present, the mainstream open-source video generation models are mainly divided into two types of architectures: MMDiT (e.g., CogVideo-X [2] and HunyuanVideo [3], which are used in the paper's experiments) and DiT (e.g., Wan2.1 [1]) architectures.

MMDiT architecture uses only a single 3D-Full Attention, which directly applies global attention to visual and text tokens. Therefore, we directly use the results of 3D attention to calculate the attention score in the proposed Attention-guided Sparse Token Distillation (introduced in Sec. 3.3). This is also the implementation in the paper's experimental part.

The DiT architecture applies two different attention mechanisms, self-attention and cross-attention. However, in the current mainstream architecture (e.g., Wan2.1 [1]), text tokens are only injected as fixed conditions, meaning that the output token of the attention block only contains visual tokens. The purpose of our proposed Attention-guided Sparse Token Distillation is to reweight the output token's contribution in the alignment to the Full-Precision model. Therefore, we only use the attention score of self-attention mechanism to calculate token-wise distribution mentioned in Eq. 11 (Line 178).

Reference:

[1]. Wan, et al. Wan: Open and advanced large-scale video generative models. Arxiv 2025.

[2]. Yang, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. Arxiv 2024.

[3]. Kong, et al. Hunyuanvideo: A systematic framework for large video generative models. Arxiv 2024.

Thank you for reviewing our manuscript and providing valuable suggestions. Here are our responses to some of the concerns you raised:

Q1: While the paper mentions that QAT requires extensive training time and large datasets, a more direct comparison or discussion on scenarios where QAT might still be preferred could strengthen the argument for PTQ.

A1: QAT conducts full training on the model weight and quantization parameters. which can guarantee relatively good performance under some extreme bits (1-2 bits). However, compared with the PTQ paradigm, QAT usually requires more training resources. For example, we compare our PTQ method with the existing QAT work for diffusion models [1][2] and report the required training resources:

Compared with the QAT paradigm, the PTQ paradigm we use can greatly reduce computing resources and calibrate larger models, which fully reflects its efficiency. It is still worth mentioning that under extreme bit (1-2 bit) quantization, QAT is still a reliable method that can guarantee the maximum performance of the model. However, in this paper, we mainly explore 4-bit quantization, and only an efficient PTQ process can ensure the nearly lossless model performance.

Reference:

[1]. Zheng, et al. Bidm: Pushing the limit of quantization for diffusion models. NeurIPS 2024.

[2]. Zheng, et al. Binarydm: Accurate weight binarization for efficient diffusion models. ICLR 2025.

[3]. Rombach, et al. High-resolution image synthesis with latent diffusion models. CVPR 2022.

Q2: Although the paper highlights a 1.3x inference acceleration, the detailed latency numbers in Table 3 show a more modest improvement, especially for the Hunyuan Video model (191.3s for FP vs. 175.2s for S²c-VDIT). This translates to approximately a 1.09x acceleration for Hunyuan Video and 1.28x for Cog VideoX-5B (259.2s for FP vs. 203.2s for S²-VDIT). Given the significant model compression (3.9x), the observed inference acceleration is not as substantial as might be expected, which could make the quantization less convincing for applications prioritizing runtime speed above all else

A2: Our empirical observation shows that HunyuanVideo does have a relatively low acceleration ratio. However, our quantization method has significant memory compression and inference acceleration across a variety of popular video generation models, and the acceleration effect is more obvious in high-resolution generation.

For the deployed quantization model, compared with CogVideo-X, we do observe that HunyuanVideo has a relatively lower inference acceleration rate. We attribute this to the dual-stream DiT block used in HunyuanVideo. The dual-stream DiT block introduces much more high-precision normalization, scale, and shift operations that cannot fully benefit from quantization. We also observed that for models that support multi-resolution generation like HunyuanVideo and Wan2.1 [1], higher resolutions that are closer to the business scenarios can bring better acceleration effects. We present the comprehensive evaluation results here:

We can see that latency results are consistent with our observations. Also on the four popular open-source video generation models, quantization can bring continuous inference acceleration. The benefits of quantization are not only reflected in latency. The compression of model storage can greatly reduce the model storage occupation. The significant reduction of inference memory can make it more widely used on devices with limited computing resources, which has high research value for the wide application of video generation models.

Reference:

[1]. Wan, et al. Wan: Open and advanced large-scale video generative models. Arxiv 2025.

Thank you very much for your high recognition of our work and the valuable suggestions you provided. Our response is as follows:

Q1: Providing more low-level metrics for video evaluation like PSNR and SSIM would further strength the superiority of the proposed methods.

A1: We provide more low-level metrics (PSNR, SSIM, and LPIPS) comparison results of CogVideoX-5B and HunyuanVideo under W4A6 setting. Compared with the existing state-of-the-art methods Quarot [1] and ViDiT-Q [2], our method achieves the best performance in all metrics.

We provide the results in the following table. The experimental results show that our method not only has better results on the VBench (Tab.1 and Tab.2) and EvalCrafter (Tab. 5) provided in the main paper, but also has advantages on the low-level metrics. It further proves that our method can generally improve the model performance.

Reference:

[1]. Ashkboos, et al. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. NeurIPS 2024.

[2]. Zhao, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. ICLR 2025.

Q2: Providing more acceleration results under different resolutions of HunyuanVideo would be better for verifying the acceleration effect in different scenarios.

A2: We provide multi-resolution efficiency study on HunyuanVideo under W4A4 setting. Compared with FP model, our method achieves obvious model compression and inference acceleration under various resolutions.

We provide the detailed results in the following table:

We find that the model acceleration effect is more obvious when the resolution is increased. We believe that when the resolution is improved and the number of tokens to be processed by the model is increased, the greater computational burden can benefit more from the quantization technique. This fully reflects the great potential of quantization for generating high-resolution video.

We are deeply grateful for your support of our work, and we provide detailed responses to your comments as follows:

Q1: It seems that the calculation of the Hessian approximation introduces non-negligible computational overhead, which can become significant as the number of calibration samples or timesteps increases.

A1: We reported the time increase caused by using the Hessian approximation when constructing the calibration dataset in different models. Compared with without using Hessian, using Hessian approximation will only bring less than 0.1% additional time, which is very efficient.

We provide the detailed comparison in the following table:

It can be seen that the computational burden of using Hessian approximation is minor, but it can bring significant performance improvement. We use the Levenberg-Marquardt approximation [1][2] to calculate the Hessian approximation, which requires only one step matrix multiplication (See Eq. 6) to obtain the approximate result, and is very efficient.

Reference:

[1]. Frantar, et al. Optimal brain compression: A framework for accurate post-training quantization and pruning. NeurIPS 2022.

[2]. Frantar, et al. Gptq: Accurate post-training quantization for generative pre-trained transformers. ICLR 2023.

Q2: The paper lacks a discussion or ablation study on the effect of calibration data size. It does not investigate how performance varies with different numbers of calibration samples, nor does it explore the trade-off between calibration data size and quantization quality.

A2: We conduct ablation study on calibration data size in CogVideoX-2B under W4A6 setting. Increasing data size will bring additional calibration burden, but it does not always bring significant performance improvement. We finally chose a compromise solution of 40 data size.

We provide the detailed results in the following table:

It can be seen that the calibration time increases almost linearly with the increase of data size. The performance of 40 data is significantly better than that of 20 data, but the performance improvement of 80 data is minor. Therefore, in the trade-off of performance and calibration time, we choose to use 40 data as the unified experimental settings, as we described in Appendix Sec. A (Line 441).