Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers

We thank the reviewer for the constructive comments on our paper. Regarding the concerns, we provide the following responses.

Q1: Optimization details.

Sorry for the misunderstanding, we have reported optimization details including training data in Appendix Sec. B. and show the training cost in Tab. 4. We will make this more prominent in revised version.

Q2: Original videos.

We apologize for our negligence and we have released all original videos in [https://anonymous.4open.science/r/Generated_videos]. We also have shown more generated critical video frames under W3A6 setting comparison with FP model in Appendix Sec. H. In W3A6 setting, current methods cannot even produce meaningful videos. Our method is significantly closer to FP in terms of generation quality than existing methods.

Q3: Some temporal flicking.

The OpenSora model we experimented in paper inevitably has some temporary flicking. But our method is significantly better than existing methods in terms of metrics (see Tab. 1 and Tab. 2) and closer to the FP model in terms of visual effects (see Fig. 5, Appendix Sec. H). We further conducted W4A6 quantization experiments on larger models HunyuanVideo and CogVideoX, and demonstrated the generated video effects in [https://anonymous.4open.science/r/Generated_videos]. The videos we generate do not have issues such as temporary flicking and the visual quality is closer to the full precision model. We also has notable improvement compared to the baseline method ViDiT-Q.

We also present quantitative comparisons in the table below：

Our method is closer to the FP model in terms of metrics and visual effects, and has a significant improvement compared to the baseline method ViDiT-Q.

Q4: Performance of our quantized model compared with FP model.

1. Some performance gap under lower bit setting. We have reported the same higher bit setting (e.g. W6A6) with ViDiT-Q in Appendix Tab. 5 and our method can achieve lossless performance compared with FP model. Since the performance gap is minor in higher bit settings, we further explore the performance improvement on lower bit settings (e.g., W3A6). Under this low bit setting, we have achieved state-of-the-art and significantly outperform existing methods and existing methods can hardly generate reasonable videos under low bit setting. We show more visual comparison in Fig. 5 and Appendix Sec. H.

2. Choice for mainly focus on lower bit. Naturally, lower bit quantization brings more memory saving and acceleration for real-world deployment, but often faces more severe performance degradation which is a harder situation under exploration. Since existing methods like ViDiT-Q have achieved almost lossless performance at higher bits (e.g., W4A8), we want to further explore the performance improvement at lower bits. Compared to W4 quantization, W3 usually faces severe performance degradation which is commonly discovered in LLM quantization [1][2]. So we chose lower bit quantization settings (e.g., W3A6) in Tab. 1 and Tab. 2, under which existing methods can hardly generate reasonable videos as shown in Fig. 5. Our method has greatly improved in terms of metrics and visual effects compared to existing methods. We want to note that we also reported same higher bit settings as ViDiT-Q in Appendix Tab. 5, and our method still has improvement compared to existing methods and achieves lossless.

[1].GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.

[2].Quarot: Outlier-free 4-bit inference in rotated llms.

3. Main contribution. We would like to further highlight our contribution by investigating the limitations of current quantization methods in video generation models and solving them from both quantization (Sec. 3.2) and optimization (Sec. 3.3) perspectives. With almost the same calibration cost with current methods (Tab. 4), our method brings significant relatively performance improvement at lower bit compared to existing methods in terms of metrics (Tab. 1 and Tab. 2) and visual effects (Fig. 5 and Appendix Sec. H) while maintaining lossless performance at higher bit (Appendix Tab. 5). Meanwhile, our method can bring 2.4x reduction in memory cost and 1.35x actual acceleration for inference (Appendix Tab. 7) with no extra burden compared with baseline ViDiT-Q.

Thank you very much for your high recognition of our work and the valuable suggestions you provided. Our response is as follows:

Q1: Quantitative analysis on quantization error.

We add quantitative experiments on W3A6 model last layer weight's quantization error and information entropy mentioned in the proposed TQE (Sec. 3.2)：

Consistent with our claim, our method indeed reduces quantization errors and improves the information entropy of the quantized weights. From quantitative metrics, TQE has indeed improved the performance of the model by reducing quantization error and increasing entropy. Theorem 3.2 states that the quantization error of weights has lower information entropy compared to the original weights. This is also proved by the quantitative results.

Q2: Actual inference efficiency and memory consumption.

We have reported the inference efficiency of W4A8 model compared with FP model in Appendix Tab. 7. We also display the data here.

Compared to FP model, our method achieves complete lossless performance while bringing 2.4$\times$ reduction in memory cost and 1.35$\times$ inference acceleration. In the revised version, we will modify our layout to make this more prominent.

Q3: In Tab. 4, why ViDiT-Q has training cost?

ViDiT-Q requires calculating quantization sensitivity between layers to allocate different bit-widths, which incurs additional time consumption. This is also reported in their paper. We will modify our wording to avoid ambiguity.

Q4: Fig.2 misses some notations.

We will fix the problem in the final version.

Q5: Similarity function used in Eq. 9.

We use cosine similarity.

Thank you for your detailed review of our work. Here are our responses to your concerns:

Q1: Statement of our Q-VDiT.

We apologize for the misunderstanding caused by our statement. We absolutely do not deny ViDiT-Q's contribution. ViDiT-Q is the first method to explore quantization for video generation models and an important baseline method for our paper. We greatly appreciate the contribution of ViDiT-Q. What the statement wants to emphasize is that we hope to consider the correlation between different frames in the video information during the optimization process and further improve the quality of the quantized video generation model. We will modify our wording in the revised version to avoid this ambiguity.

Q2: Quantitative analysis on quantization error and information entropy introduced in TQE.

We add quantitative experiments on W3A6 model last layer weight's quantization error and information entropy mentioned in the proposed TQE (Sec. 3.2)：

Consistent with our claim, our method indeed reduces quantization errors and improves the information entropy of the quantized weights. From quantitative metrics, TQE has indeed improved the performance of the model by reducing quantization error and increasing entropy. Theorem 3.2 states that the quantization error of weights has lower information entropy compared to the original weights. This is also proved by the quantitative results.

Q3: Explanation of quantization settings.

Naturally, lower bit quantization brings more memory saving and acceleration for real-world deployment, but often faces more severe performance degradation, which is a harder situation under exploration. Since existing methods like ViDiT-Q have achieved almost lossless performance at higher bits (e.g., W4A8), we want to further explore the performance improvement at lower bits. Compared to W4 quantization, W3 usually faces severe performance degradation, which is commonly discovered in LLM quantization [1][2]. So we chose lower bit quantization settings (e.g., W3A6) in Tab. 1 and Tab. 2, under which existing methods can hardly generate reasonable videos as shown in Fig. 5. Our method has greatly improved in terms of metrics and visual effects compared to existing methods. We want to note that we also reported the same higher bit settings as ViDiT-Q in Appendix Tab. 5, and our method still has improvement compared to existing methods and achieves lossless.

[1].GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.

[2].Quarot: Outlier-free 4-bit inference in rotated llms.

Q4: Performance gap between W3A8 and W4A8.

Compared to W4, the significant loss of weight information in W3 (only 2**3=8 representation candidates for a single value) leads to a serious performance degradation as we discussed in Q3. This also highlights the importance of our proposed TQE in compensating for the weight quantization errors. Our method outperforms current quantization methods in both W3 and W4 (Tab. 1 and Tab. 2).

Q5: Motion dynamics before and after quantization.

Existing methods like ViDiT-Q have also found that the quantized model exhibits a certain degree of dynamic decline. We can also see from the metrics Motion Smoothness and Dynamic Degree in Tab. 1 that this is particularly severe after W3 quantization. Our method has the same level of motion dynamics as the full precision model in terms of metrics, which can be seen from the Motion Smoothness and Dynamic Degree metrics in Tab. 1. And our method also shows significant improvement in dynamics compared to existing methods. More generated videos in [https://anonymous.4open.science/r/Generated_videos] also prove the dynamic retention ability of our method at the same level as the FP model.

Q6: Evaluation on larger video generation models.

We add W4A6 quantization experiment on larger SOTA models CogVideoX-5B and HunyuanVideo-13B:

We also provide more generated video comparisons in [https://anonymous.4open.science/r/Generated_videos]. Our method is closer to the FP model in terms of metrics and visual effects, and has notable improvement compared to the baseline method ViDiT-Q.

Thank you for reviewing our manuscript and providing valuable suggestions. Here are our responses to some of the concerns you raised:

Q1: Original model’s qualitative results.

We apologize for our negligence and we have released all original videos in [https://anonymous.4open.science/r/Generated_videos]. We have also shown more generated critical video frames under W3A6 setting comparison with FP model in Appendix Sec. H. In W3A6 setting, current methods cannot even produce meaningful videos. Our method is significantly closer to FP model in terms of generation quality than existing methods.

Q2: Overall quality improvement.

We have reported the same higher bit setting (e.g. W6A6) with ViDiT-Q in Appendix Tab. 5 and our method can achieve lossless performance compared with FP model. Since the performance gap is minor in higher bit settings, we further explore the performance improvement on lower bit settings (e.g., W3A6). Under this low bit setting, we have achieved state-of-the-art and significantly outperform existing methods and existing methods can hardly generate reasonable videos under low bit setting. We show more visual comparison in Fig. 5 and Appendix Sec. H.

We further conducted W4A6 quantization experiments on larger models HunyuanVideo and CogVideoX, and demonstrated the generated video effects in [https://anonymous.4open.science/r/Generated_videos]. The visual quality of our method is closer to the full precision model. We obtained better results than baseline method ViDiT-Q.

We also present quantitative comparisons in the table below：

Our method is closer to the FP model in terms of metrics and visual effects, and has a significant improvement compared to the baseline method ViDiT-Q.

Q3: Additional lower-level metrics (e.g., FVD).

We have reported FVD on UCF-101 dataset in Appendix Tab. 6. We also report FVD on OpenSORA prompt set used in Tab. 1 and Tab. 2:

Compared to baseline method ViDiT-Q, our method achieved lower FVD, demonstrating the consistent superiority of our method on both high and low level metrics.

Q4: Inference speed comparison.

We have reported the inference efficiency of W4A8 model compared with FP model in Appendix Tab. 7. We also display the data here.

Compared to FP model, our method achieves complete lossless performance while bringing 2.4$\times$ reduction in memory cost and 1.35$\times$ inference acceleration. In the revised version, we will modify our layout to make this more prominent.

Q5: Motivation for temporal distillation.

We quantitatively compare our distillation method with temporal differences:

As we discussed in Sec. 3.3, our motivation is to perceive the inter-frame information of the whole video in distillation process. We hope to consider the overall information of the video while optimizing single frame information. But direct MSE calculates different frame information separately as shown in Eq. 12. Although temporal differences can improve performance to some extent, our method achieved better results. Concatenating all frames together can directly model the information of different frames in the whole video, which actually includes temporal differences between frames. Our method can directly model the relationship matrix between all frames instead of simply the gap between two frames, so it can capture the global optimization information of the video well and achieve better performance.