VETA-DiT: Variance-Equalized and Temporally Adaptive Quantization for Efficient 4-bit Diffusion Transformers

Thank you sincerely for your thoughtful and positive feedback on our work. Below, we have provided a detailed explanation for your remaining concern as follows. Please do not hesitate to let us know if you have any further questions.

W1 & Q1: Hardware resource usage evaluation.

We thank the reviewer for pointing this out. We have updated the manuscript to include an evaluation of hardware resource usage. In DiTs, linear layers dominate both computation and memory consumption [1, 2]. To address this, we implement our INT4 matrix multiplications using CUTLASS on Tensor Cores (CUDA 12.4) and evaluate our custom kernels on NVIDIA A100 GPU.

We fuse most KLT matrices into model weights during offline calibration, thereby eliminating the need for runtime transformation. For layers that require runtime KLT to preserve mathematical equivalence [3], we use fast online Hadamard kernels, which are widely available and efficient.

Despite incorporating both online quantization and runtime KLT transforms, we still observe 1.95$\times$ speedup and 3.74$\times$ memory savings over FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides 1.23$\times$ speedup and 1.81$\times$ memory reduction, demonstrating the hardware efficiency of our method:

Bit-Width	Method	Latency (ms)	Speedup	Memory (MB)	Reduction
W16A16	FP	6.167 ± 0.045	-	38.53	-
W4A8	ViDiT-Q	3.879 ± 0.012	1.59$\times$	18.68	2.06$\times$
W4A4	Ours	2.960 ± 0.006	2.08$\times$	9.68	3.98$\times$
W4A4 + Online Transform	Ours	3.163 ± 0.007	1.95$\times$	10.31	3.74$\times$

We believe this provides a comprehensive view of both computational and memory efficiency. We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.

[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.

[2] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.

[3] SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR 2024.

W2 & Q2: Evaluation under lower bit settings.

We conducted additional experiments on DiT-XL/2 under the W3A3 quantization setting, which is already highly challenging. As shown below, our method significantly outperforms Q-DiT, but performance degradation is still notable under such low bit-widths. Introducing mixed-precision, by keeping some sensitive layers in 8-bit, yields substantial quality improvements.

Bit-Width	Method	IS $\uparrow$	FID $\downarrow$	sFID $\downarrow$	Precision $\uparrow$
W16A16	FP	164.61	4.86	17.65	0.80
W3A3	Q-DiT	1.65	266.97	402.47	0.01
W3A3	Ours	45.83	55.42	34.42	0.32
W3A3(Mixed Precision)	Ours	111.62	13.71	22.25	0.66

At even lower bit-widths, quantizing becomes extremely lossy due to limited representational range. In such cases, offline KLT-based variance equalization becomes insufficient, and online KLT computation may be required to maintain generative fidelity. However, this would introduce non-negligible runtime cost. We will explore this trade-off in future work.

We greatly appreciate the reviewer's constructive comments on our paper. We will respond to the reviewer's feedback with detailed explanations for each point.

W1 & Q1: Difference from MambaQuant in the use of KLT.

We also noticed that MambaQuant applies KLT to state space models, as described in line 147 of our paper. MambaQuant focuses on Mamba-based state space models and addresses the numerical instability caused by outlier amplification in PScan operations. In contrast, our work is dedicated to Transformer-based diffusion models (DiTs), which exhibit a naturally large variance across activation dimensions and significant temporal variation due to iterative denoising.

We introduce KLT to equalize the intrinsic variance distribution across channels and time steps in DiTs. However, applying a single, time-step-independent KLT matrix across all denoising steps, as MambaQuant, leads to a severe degradation in generation quality due to the significant temporal variation in DiT activations. On the other hand, using fully time-step-specific KLT matrices incurs excessive load/offload overhead during inference. To address this trade-off, we propose an incoherence-aware adaptation strategy that enables efficient and temporally-sensitive KLT usage in DiTs, effectively mitigating both representational and computational inefficiencies.

We will include a more explicit discussion of the differences between our approach and MambaQuant in the revised manuscript.

W2: Effectiveness compared to alternative temporal sampling strategies.

Prior methods, such as ViDiT-Q [1], adopt a shared balancing mask $s$ across all denoising steps, which fails to account for the dynamically changing data distributions inherent in DiTs, effectively corresponding to uniform timestep sampling without adaptation. Currently, there is a lack of viable strategies specifically designed to handle this temporal variation.

Therefore, we provide a comprehensive evaluation of our incoherence-aware adaptation in Table 3 and Appendix D.2 of our paper, assessing both its impact on final generation quality and its effectiveness in reducing quantization error during inference. The results demonstrate that our method significantly outperforms uniform and random sampling strategies.

[1] ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation. ICLR 2025.

W3 & Q4: Runtime and calibration overhead.

The calibration process is conducted once offline and does not affect inference latency. On NVIDIA A100 GPUs, calibrating DiT-XL/2 takes approximately 35 minutes, and calibrating Open-Sora v1.2 (1.1B parameters) takes around 2.5 hours.

During inference, we use CUTLASS INT4 kernels optimized for Tensor Cores (CUDA 12.4) and fuse most KLT matrices into the model weights. Only a few blocks (e.g., out-proj, down-proj) require lightweight online transforms, implemented via fast Hadamard kernels commonly available in existing libraries. We still achieve 1.95$\times$ speedup and 3.74$\times$ memory savings over FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides 1.23$\times$ speedup and 1.81$\times$ memory reduction, demonstrating the hardware efficiency of our method:

Bit-Width	Method	Latency (ms)	Speedup	Memory (MB)	Reduction
W16A16	FP	6.167 ± 0.045	-	38.53	-
W4A8	ViDiT-Q	3.879 ± 0.012	1.59$\times$	18.68	2.06$\times$
W4A4	Ours	2.960 ± 0.006	2.08$\times$	9.68	3.98$\times$
W4A4 + Online Transform	Ours	3.163 ± 0.007	1.95$\times$	10.31	3.74$\times$

These results confirm that our method maintains efficiency while incorporating KLT-H adaptations. We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.

W4 & Q2: Temporal incoherence in video generation.

Temporal incoherence also arises in video synthesis. We analyzed activation statistics at different denoising steps in Open-Sora and observed similar patterns of variation as in image generation. The table below presents varying incoherence values across timesteps, confirming the presence of temporal drift in diffusion-based video generation:

Timestep	Mean	Max	Incoherence
t1	-0.00414	3.201	10.821
t5	-0.00140	3.262	11.109
t10	0.00042	3.283	13.344
t15	0.00073	3.408	11.008
t20	0.00108	3.773	12.742
t25	0.00117	4.191	14.078
t30	0.00267	5.391	15.906

As shown in Table 2 of our paper, our incoherence-aware adaptation continues to deliver substantial improvements in video generation quality, confirming its effectiveness in modeling temporal variations in both image and video diffusion models.

Q3: Effectiveness and novelty of incoherence-aware adaptation.

MambaQuant applies a shared KLT transform across all time steps. However, these assumptions break down in DiTs, where iterative denoising introduces highly dynamic and non-stationary activation patterns.

Our incoherence-aware adaptation departs from this assumption by designing a temporally-sensitive calibration strategy specifically tailored for DiTs. Rather than relying on handcrafted or static transforms, we leverage synthetic data generation to adaptively construct calibration sets that capture evolving activation statistics across time steps—a capability absent in prior KLT-Hadamard quantization methods.

Moreover, our approach precomputes transform matrices offline and fuses them into model weights wherever possible, minimizing runtime overhead without sacrificing temporal sensitivity.

This combination of temporal modeling, data-driven calibration, and runtime efficiency distinguishes our method both in novelty and practical utility.

Q5: Mixed-precision considerations in comparison with ViDiT-Q.

In the official implementation of ViDiT-Q, certain Linear layers are retained in FP16, which substantially reduces quantization-induced degradation but also limits efficiency gains, especially since linear layers are bottlenecks in DiTs [1, 2]. For a fair comparison, we disabled mixed-precision in ViDiT-Q.

To test whether our method can benefit similarly, we evaluated W3A3 quantization on DiT-XL/2 with and without mixed precision. Although W3A3 is a highly aggressive setting, our method benefits significantly from a mixed-precision strategy:

Bit-Width	IS $\uparrow$	FID $\downarrow$	sFID $\downarrow$	Precision $\uparrow$
W16A16	164.61	4.86	17.65	0.80
W3A3	45.83	55.42	34.42	0.32
W3A3(Mixed Precision)	118.62	13.71	22.25	0.66

This indicates that VETA-DiT is also compatible with hybrid-precision schemes. However, as in ViDiT-Q, these approaches reduce theoretical efficiency gains and often require specialized CUDA kernels or hardware to realize end-to-end improvements.

[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.

[2] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.

We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.

W1 & Q1: Efficiency metrics.

We have extended the manuscript to include latency and memory comparisons. Our implementation of VETA-DiT is built upon CUDA 12.4 with CUTLASS to support INT4 matrix multiplication on Tensor Cores. We evaluated the performance of the linear layers on an NVIDIA A100 GPU, which are the main bottlenecks in DiT [1, 2].

Most KLT matrices are fused into model weights to eliminate runtime overhead. For layers such as out-proj and down-proj, we apply online transformations using fast Hadamard kernels. Even with online quantization and KLT, our method still achieves 1.95$\times$ speedup and 3.74$\times$ memory savings compared to FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides 1.23$\times$ speedup and 1.81$\times$ memory reduction, highlighting its superior efficiency:

Bit-Width	Method	Latency (ms)	Speedup	Memory (MB)	Reduction
W16A16	FP	6.167 ± 0.045	-	38.53	-
W4A8	ViDiT-Q	3.879 ± 0.012	1.59$\times$	18.68	2.06$\times$
W4A4	Ours	2.960 ± 0.006	2.08$\times$	9.68	3.98$\times$
W4A4 + Online Transform	Ours	3.163 ± 0.007	1.95$\times$	10.31	3.74$\times$

We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.

[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.

[2] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.

W2 & Q2: Effectiveness with distilled timesteps.

While methods such as DMD reduce the number of denoising steps, temporal variation in activation statistics still exists, which is central to our approach. We analyze activations from randomly selected Transformer blocks and observe notable differences in mean, max, and incoherence across timesteps:

Timestep	Mean	Max	Incoherence
t1	-0.000235	4.200	5.680
t2	-0.000419	4.375	5.871
t3	-0.000179	4.956	4.316
t4	-0.000573	5.363	7.242

Our incoherence-aware adaptation is designed to capture and address such temporal variation, regardless of the total number of steps. We quantized the Transformer blocks in the DMD2-SDXL-4step model and evaluated on COCO. Our method achieves better results than Q-DiT under lower bit-widths:

Bit-Width	Method	IS $\uparrow$	FID $\downarrow$	sFID $\downarrow$	CLIP $\uparrow$	IR $\uparrow$
W16A16	FP	42.39	54.64	263.93	0.27	0.89
W4A8	Q-DiT	41.67	56.80	274.34	0.27	0.90
W4A8	Ours	41.40	54.89	264.86	0.26	0.88
W4A4	Q-DiT	32.93	74.19	293.94	0.23	0.43
W4A4	Ours	40.14	58.39	277.15	0.25	0.87

Our approach can be effectively combined with timestep distillation techniques like DMD without additional runtime cost, thus improving efficiency.

W3 & Q2: Scalability to larger models.

The Open-Sora v1.2 model contains 1.1B parameters, which is comparable in scale to Wan2.1-T2V-1.3B. For even larger models like Wan2.1-VACE-14B, which adopt deeper Transformer stacks, the architecture remains consistent across layers.

It has been observed that larger models are typically more robust to PTQ degradation [3]. We conducted a block-wise MSE analysis on Wan2.1-14B, and our method shows significantly lower quantization error compared to RTN and Q-DiT:

Bit-Width	MSE(RTN)	MSE(Q-DiT)	MSE(Ours)
W4A8	0.02546	0.00766	0.00374
W4A4	0.07424	0.01530	0.00714

[3] Scaling Laws for Precision. ICLR 2025.

W4 & Q3: Calibration cost of KLT.

The KLT calibration is a one-time offline process and does not incurs inference-time overhead. We report the calibration time on NVIDIA A100 GPU:

• DiT-XL/2: ~35 minutes
• Open-Sora v1.2 (1.1B): ~2.5 hours

Thanks for your time in dealing with our work. We will answer the question and discuss point by point as follows. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.

W1 & Q3: Generalization concern with calibration data.

KLT is indeed data-dependent. We address this concern by employing an incoherence-aware adaptation strategy to construct the calibration set for guiding the KLT process. Details regarding the calibration set composition and settings are provided in Appendix. A of our paper. Specifically:

• For the DiT-XL/2 model, we randomly select 16 classes from ImageNet for calibration, and perform generation across all 1000 classes.
• For PixArt-$\Sigma$ and Open-Sora models, we adopt prompts provided in their official toolkits for calibration and conduct generation on COCO and VBench datasets, which are not used during calibration.

We empirically validate that this strategy ensures the learned KLT transform generalizes across different data distributions in Section 4.2 of our paper. To further verify its generalization, we evaluate the variance-equalization effect both within and outside the calibration set:

Dataset Scope	Var(Before Transform)	Var(After Transform)
In-Distribution	0.2328 ± 0.0006	0.0885 ± 0.0001
Out-of-Distribution	0.3275 ± 0.0035	0.1009 ± 0.0003

The demonstrate show that even on out-of-distribution data, our method significantly reduces variance, indicating its strong generalization capability.

Q1: Inference overhead from KLT-Hadamard transform.

The KLT-enhanced Hadamard matrix is defined as $\mathbf{T}_{ \text{KLT-H}} = \mathbf{KH}$, where $\mathbf{K}$ is a data-driven KLT matrix, precomputed offline using the calibration set, and $\mathbf{H}$ is a random Hadamard matrix.

During inference, this transformation can be fused into model weights, incurring no additional cost for most layers. For layers like out-proj and down-proj, online transformation is necessary to preserve computational invariance [1]. We evaluated the associated overhead on DiT-XL/2 via implementing a custom CUDA kernel on NVIDIA A100 GPU, targeting matrix multiplication operations within these layers.

Bit-Width	Method	Latency (ms)	Speedup	Memory (MB)	Reduction
W16A16	FP	6.167 ± 0.045	-	38.53	-
W4A8	ViDiT-Q	3.879 ± 0.012	1.59$\times$	18.68	2.06$\times$
W4A4	Ours	2.960 ± 0.006	2.08$\times$	9.68	3.98$\times$
W4A4 + Online Transform	Ours	3.163 ± 0.007	1.95$\times$	10.31	3.74$\times$

Due to the availability of efficient Hadamard transform implementations, the online transform introduces only 6.8% overhead, while still achieving 1.95$\times$ acceleration and 3.74$\times$ memory savings over FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides 1.23$\times$ speedup and 1.81$\times$ memory reduction, highlighting its superior efficiency. We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.

[1] SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR 2024.

Q2: Sensitivity to timestep importance estimation.

Section 4.3 presents our evaluation of the sensitivity to different timestep-wise importance scoring strategies. Table 3 shows that assigning uniform scores across timesteps significantly degrades performance (FID increases from 4.86 to 48.32), leading to severe degradation in image quality. In contrast, applying our incoherence-aware adaptation reduces FID to 7.90, substantially mitigating performance loss.

Additionally, Appendix D.2 demonstrates that our strategy consistently outperforms random timestep sampling in reducing quantization error.

Q4: Challenges in extending to lower bit-widths.

We conducted generation experiments on DiT-XL/2 under the challenging W3A3 setting. We summarize the results in the table below:

Bit-Width	Method	IS $\uparrow$	FID $\downarrow$	sFID$\downarrow$	Precision$\uparrow$
W16A16	FP	164.61	4.86	17.65	0.80
W3A3	Q-DiT	1.65	266.97	402.47	0.01
W3A3	Ours	45.83	55.42	34.42	0.32
W3A3(Mixed Precision)	Ours	118.62	13.71	22.25	0.66

Although our method significantly outperforms Q-DiT, performance degradation is still observed under such extreme quantization. To mitigate this, we maintain 8-bit precision in sensitive layers, achieving better results in mixed-precision settings.

Looking forward, as bit-widths decrease further (e.g., to 2-bit), offline KLT becomes insufficient due to severely limited data representation. Online KLT may be required to dynamically equalize variance during inference, but this would incur non-trivial computational cost, which we plan to explore in future work.