SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models

Strong Statements

"Weight-only quantization cannot accelerate diffusion models" - For modern GPUs, maybe, but this lacks concrete evidence. Also, different hardware platforms have different bottlenecks.

We are referring to modern GPUs. As shown in Figure 8 of our original submission, NF4 weight-only quantization achieves the same latency as the original BF16 model on desktop-level 4090 GPUs. Similarly, Figure 3 in QServe [1] shows that W4A16 (green lines) and W16A16 (red lines) converge to the same peak performance at large batch sizes. Diffusion models process thousands of tokens in parallel, effectively resulting in a large batch size. We have updated our manuscript to provide this GPU context.

"Weights and activations must be quantized to the same bit width" - Can custom hardware support direct mixed precision operations?

Yes, specialized hardware may support mixed-precision operations. We have revised our manuscript to limit the scope to modern GPUs. Thanks for your feedback.

"they primarily consider weight-only quantization..." For example, the authors have cited Q-diffusion and EfficientDM which quantize both activations and weights. This statement needs further justification.

Thanks for your feedback. We refer to methods like QLoRA [2] here, which fine-tune LoRA branches on a quantized base model. Q-Diffusion was already cited earlier in the previous paragraph, so it was not included here. We've revised our manuscript to clarify this claim.

Computation Flow

In our original submission, we have provided an overview of our method in Figure 3 and illustrated the computation flow in Figure 6(b). Could you advise on any additional computation flows that should be included in the manuscript?

Comparison with LoRC

In our original submission, we discussed LoRC in Section 2 (Low-rank decomposition) and compared it in Section 5.2 (Ablation study) and Figure 9. We highlight the results below. Our SVDQuant significantly outperformed LoRC. Specifically, LoRC uses a low-rank branch to compensate quantization errors, but this approach is suboptimal. Quantization errors resemble white noise with a smooth singular value distribution, making low-rank compensation less effective. In contrast, SVDQuant decomposes weights and quantizes only the residual. As shown in Figure 5, the first few singular values dominate, enabling us to shift them to the low-rank branch. This reduces weight magnitude and eases the weight quantization. Furthermore, by migrating activation outliers into the weights, SVDQuant lowers activation quantization difficulty, resulting in better image quality.

Clarification on the Upcast

This is a misunderstanding. The sentence refers to quantization without the low-rank branch, where "side" refers to either weights or activations. For example, in W4A16 weight-only quantization on GPUs, weights are upcast to 16 bits during computation, meaning the operations still use 16-bit multiplication. We have revised our manuscript to make this clearer.

Clarification on Figure 3

Quantization level refers to the number of distinct values that can be represented. For example, 3-bit quantization allows for 8 distinct values, so our figures show 8 level lines. The top-left numbers indicate the largest representable values, which can be fractional numbers. After smoothing, the peak value of $|{\hat{\boldsymbol{W}}}|$ should be 4.5, not 0.5—this was a typo, and we have corrected it in our revision. Yes, smaller peaks mean easier to quantize.

QKV Projections

Thank you for your suggestion. Recent diffusion models include transformer blocks to enhance performance. The QKV projections refer to the layers within these transformer blocks. We have updated our manuscript to clarify this concept.

References

1. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
2. QLoRA: Efficient Finetuning of Quantized LLMs

Thanks for supporting our paper. Below, we address each of your questions individually.

Lambda Selection

The smoothing factor $\lambda \in \mathbb R^{m}$ is a per-channel vector whose $i$-th element can be computed as $\lambda_i = \max(|{\boldsymbol{X}} _{:,i}|)^\alpha / \max(|{\boldsymbol{W}} _{i,:}|)^{1-\alpha}$ following SmoothQuant [1]. Here, ${\boldsymbol{X}} \in \mathbb R^{b\times m}$ and ${\boldsymbol{W}} \in \mathbb R^{m\times n}$. It is decided offline by searching for the best migration strength $\alpha$ for each layer to minimize the layer output mean squared error (MSE) after SVD on the calibration dataset. We have included this in Appendix B.

Blackwell Performance

Blackwell GPUs were not on the market at the time of submission. However, our fused kernel is expected to achieve similar or better speedup on these GPUs. The advantage of kernel fusion lies in reducing memory access, making it independent of the specific computation instructions. As long as the ratio of peak performance to peak memory bandwidth remains unchanged, we anticipate similarly minimal overhead. Furthermore, with native microscaling group quantization support, we can anticipate greater speedups compared to our INT4 results on 4090 GPUs.

Novelty

Thanks for your interest in our method.

Speedup Compared to Pure Low-Bit Kernels

On the laptop-level 4090 GPU, naive INT4 FLUX requires 212ms for a single forward. SVDQuant introduces low-rank branches, increasing latency to 250ms without optimization (18% overhead). LoRunner effectively cuts off redundant memory access and achieves 218ms latency, adding just 3% overhead.

Thanks for raising the score. Below, we address each of your questions individually.

Bitwidth-Performance Tradeoff

We evaluate LPIPS across different bitwidths for various quantization methods on PixArt-∑ and FLUX.1-schnell using the MJHQ dataset, with weights and activations sharing the same bitwidth. Following the convention [1-6], for bitwidths above 4, we apply per-channel quantization; for 4 or below, we use per-group quantization (group size 64). SVDQuant consistently outperforms naive quantization and SmoothQuant. Notably, on PixArt-∑ and FLUX.1-schnell, our 4-bit results match 7-bit and 6-bit naive quantization, respectively. See Figure 18 in our revised manuscript for the tradeoff curve.

Low-Rank Branch Overhead

Introducing a low-rank branch significantly improves the tradeoff between image quality and bitwidth in SVDQuant. For example, achieving the same quality as SVDQuant’s W4A4 requires SmoothQuant to use approximately 6 bits as shown in the above tradeoff, resulting in a ~50% overhead. In contrast, with LoRunner, the low-rank branch adds only a 3% overhead for a single DiT forward pass (see Table 4 in our revised manuscript). Additionally, SVDQuant consistently outperforms QLLM on both PixArt-∑ and FLUX.1-schnell, with a particularly notable advantage on PixArt-∑, as shown in the below table.

Moreover, users often apply custom low-rank adapters (LoRAs) when using diffusion models. LoRunner eliminates redundant memory access for the low-rank branch, enabling it to be retained during inference. This allows seamless support to LoRAs by appending their weights to the low-rank branch, which avoids the need for weight fusion and re-quantization. Figure 9 and 17 in our revised manuscript show some visual results of applying five different LoRAs to our INT4 FLUX.1-dev, matching the image quality of the original 16-bit model.

Comparisons with LLM Baselines

NF4 is a strong baseline in terms of image quality, which uses W4A16 precision instead of W4A4. At the reviewer’s request, we adapted LLM quantization methods (QLLM, QuaRot, and AffineQuant) to diffusion models and show W4A4 results above on MJHQ. SpinQuant is not applicable due to the high overhead of rotation-based methods, as explained in Section 3 of our original submission. These methods require fusing layer norm weights into the following linear layer and fusing rotation matrix into both the previous and following linear layers. However, Diffusion Transformers (DiTs) use adaptive layer norms whose weights are dynamically generated based on the input condition. This prevents the rotation matrix from being merged into the previous linear layer. Therefore, rotation-based methods must perform online rotation, which adds an extra full-rank matrix multiplication for SpinQuant and results in a prohibitive 730% measured latency overhead on FLUX.1 models. As an alternative, we evaluate another well-known rotation-based method QuaRot, which exploits Hadamard transformation as rotation. Even optimized with the SoTA fast_hadamard_transform package, QuaRot incurs a 21% overhead, much higher than ours.

SVDQuant outperforms all baselines by a wide margin across all metrics on PixArt-∑. On FLUX.1-schnell, SVDQuant achieves the best FID and Image Reward scores and ranks second in PSNR only behind NF4.

Thanks for the general reply, and I have increased my score. Moreover, what is the setting of QuaRot in your experiments, RTN or GPTQ for weight quantization?

Thanks for supporting our paper. Below, we address each of your questions individually.

8-Bit Latency Comparison

Below we compare the FLUX latency of the 8-bit and our 4-bit models on a desktop-level 4090 GPU. Our 4-bit model achieves approximately 1.3× speedup over the 8-bit model. We have correspondingly updated the results in our revision (see Appendix D.3 and Table 4).

Comparison with Atom

As suggested by the reviewer, we compared SVDQuant with Atom on FLUX.1-schnell using the MJHQ dataset. In the INT W4A4 setting, SVDQuant outperforms Atom across all metrics. Additionally, on a desktop-level 4090 GPU, SVDQuant with LoRunner is 1.2× faster than Atom.

Smoothing Process

The smoothing factor $\lambda \in \mathbb R^{m}$ is a per-channel vector whose $i$-th element can be computed as $\lambda_i = \max(|{\boldsymbol{X_{:,i}}}|)^\alpha / \max(|{\boldsymbol{W_{i,:}}}|)^{1-\alpha}$ following SmoothQuant [1]. Here, ${\boldsymbol{X}} \in \mathbb R^{b\times m}$ and ${\boldsymbol{W}} \in \mathbb R^{m\times n}$. It is decided offline by searching for the best migration strength $\alpha$ for each layer to minimize the layer output mean squared error (MSE) after SVD on the calibration dataset. We have included this in Appendix B.

Ablation Setting

Yes, "SVD-only" refers to applying SVD to the weights and keeping only the low-rank component without quantization. "Naive quantization" refers to directly applying per-group quantization without any smoothing.

References

1. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Thanks for supporting our paper. Below, we address each of your questions individually.

Results Below 4-Bit Quantization

Below we show the results of sub-4-bit quantization. Our SVDQuant can still generate images in the 3-bit settings on both PixArt-∑ and FLUX.1-schnell, performing much better than SmoothQuant. Below this precision (e.g., W2A4 or W4A2), SVDQuant cannot produce images either, since 2-bit symmetric quantization is essentially a ternary quantization. Prior work [1,2] has shown that ternary neural networks require quantization-aware training even for weight-only quantization to adapt the weights and activations to the low-bit distribution. We have included the discussion in Appendix D.5 in our revised paper.

Comparisons with Q-Diffusion and QuEST

In our submission, we demonstrated that SVDQuant outperforms MixDQ and ViDiT-Q (see Table 1 and Figure 7). Both MixDQ and ViDiT-Q have already compared against Q-Diffusion [3] as a baseline and outperformed it in their respective papers (refer to Figure 1, Tables 1 and 2 in MixDQ, and Table 3 in ViDiT-Q). QuEST [4] is a quantization-aware training method that requires additional fine-tuning and was therefore not included in our submission.

At the reviewer's request, we conducted additional experiments comparing these baselines on Stable Diffusion 1.4 using the MJHQ dataset. SVDQuant still achieves significantly better LPIPS and PSNR. We have included the results in Appendix D.3 and Table 4 in our revised manuscript.

References

1. BitNet: Scaling 1-bit Transformers for Large Language Models
2. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
3. Q-Diffusion: Quantizing Diffusion Models
4. QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning

Rank Ablation

In our original submission, we have included an ablation study on rank selection in Section 5.2 (Trade-off of increasing rank) and Figure 10. We found that a rank of 32 effectively restores the image quality of the original 16-bit model with only about 5% latency and memory overhead. While increasing the rank to 64 further improves image quality, the overhead (~10%) becomes substantial. Additionally, Figure 5 shows that the first 32 singular values of the transformed weights are significantly larger, explaining why rank 32 is optimal.

Robustness of Different Weight Distribution

Our method's robustness to non-Gaussian distributions can be demonstrated both theoretically and empirically:

• Theoretically, while Proposition 4.2 uses Gaussian assumptions for simplicity, our framework can be easily generalized to any distribution satisfying: $\mathbb{E}[\max(|{\boldsymbol{R}}|)] \le c\cdot \mathbb{E}[||{\boldsymbol{R}}||_F],$ where $c$ characterizes the distribution's skewness. The core relationship between matrix norm and quantization error holds across distributions, though the precise scaling factor may vary. See Proposition 2 and Appendix A.2 in our revised manuscript for details.
• Empirically, QLoRA [1] shows that neural network weights typically follow near-normal distributions except for a small number of outliers (see its Appendix F). Our low-rank branch helps mitigate these outliers, leaving a more regular distribution that aligns better with our theoretical assumptions.

Mixed Precision

Our LoRunner engine supports mixed precision by decoupling the precision-dependant low-bit branch computation loop and the epilogue for low-rank branches. Adapting to any other precisions only requires updating the main loop, making LoRunner highly flexible to different precision configurations.

Results below 4-Bit Quantization

As requested by the reviewer, below we show the results of sub-4-bit quantization. Our SVDQuant can still generate images in the 3-bit settings on both PixArt-∑ and FLUX.1-schnell, performing much better than SmoothQuant [7]. Below this precision (e.g., W2A4 or W4A2), SVDQuant cannot produce images either, since 2-bit symmetric quantization is essentially ternary quantization. Prior work [9,10] has shown that ternary neural networks require quantization-aware training even for weight-only quantization to adapt the weights and activations to the low-bit distribution.

Speed Breakdown

We compare the latency and memory usage of a single forward pass for INT4 FLUX.1 DiT on a 4090 desktop GPU in the below table. Without optimization, SVDQuant incurs about 18% latency overhead due to the low-rank branches. LoRunner significantly reduces this overhead, achieving a similar latency to naive INT4. Memory usage remains unaffected by LoRunner, with the low-rank branches adding only ~3% memory costs.

Typos

Thanks for pointing them out. We have fixed them in our new revision.

References

1. QLoRA: Efficient Finetuning of Quantized LLMs
2. ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
3. QNCD: Quantization Noise Correction for Diffusion Models
4. Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers
5. MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization
6. EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models
7. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
8. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
9. BitNet: Scaling 1-bit Transformers for Large Language Models
10. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Novelty & Related Work Comparisons

Previous works combining low-rank decomposition with quantization primarily focus on parameter-efficient fine-tuning (PeFT) such as QLoRA [1], where low-rank branches are fine-tuned on a quantized base model. These methods do not aim to improve quantization performance and only quantize weights. The exception, LoRC [2], uses a low-rank branch to compensate for quantization errors. However, this approach is suboptimal, as quantization errors resemble white noise, which low-rank branches struggle to mitigate effectively, as shown in Section 5.2 (Ablation study) and Figure 9 of our original submission. In contrast, we carefully analyze the sources of quantization errors and propose a principled post-training quantization (PTQ) method to absorb outliers in both weights and activations, significantly reducing errors. Additionally, our inference engine is organic with our quantization approach. Without it, the overhead from low-rank branches would negate the potential speedup, making our method impractical for real-world use on modern GPUs.

As requested by the reviewer, we discuss the similarities and differences between our method and related works, including QNCD [3] and Q-DiT [4] in Table 2 of our revised manuscript. Our method differs distinctly from these works. To avoid being too verbose, we only sample 4 representative works below.

Theoretical Analysis of Why SVDQuant Works Better

We acknowledge the desire for theoretical analysis. While a comprehensive theory for quantization methods is challenging due to dependencies on data and weight distributions, we did provide meaningful theoretical insights in our paper:

• Our Proposition 4.1 decomposes quantization error into activation and weight components, which we address through smoothing and low-rank decomposition respectively.
• Proposition 4.2 shows that quantization error scales with matrix norm. Our low-rank branch reduces this norm, minimizing error and providing a theoretical basis for our strong empirical performance.

Compared to other outlier-handling methods like smoothing and rotation:

• Smoothing shifts outliers between weights and activations, making activation quantization easier but weight quantization harder, resulting in limited overall error reduction.
• Rotation redistributes outliers across channels but fails to reduce the matrix norm, leading to higher errors than our approach.

We also provide experimental comparisons with SmoothQuant [7] (smoothing-based) and QuaRot [8] (rotation-based) on the MJHQ dataset under W4A4 settings. As shown in the table below, SVDQuant outperforms both SmoothQuant and QuaRot on PixArt-∑ and FLUX.1-schnell, particularly with a large margin on PixArt-∑.

Code

As promised in the abstract, we will release the code and models upon publication. The code is available at this anonymous link.