IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models

Results on More Diffusion Models.

Good comments! As suggested, we evaluate our IntLoRA on the FLUX.1-dev model. Since FLUX is notoriously costly to fine-tune even using LoRA, due to limited computational resources, we only give the results of FP16 vanilla LoRA and our IntLoRA on 15 text-subject pairs of Dreambooth. The results are shown below.

\begin{array}{l|ccccc} \hline \text{methods} & \text{nbits} & \text{DINO} & \text{CLIP-I} & \text{CLIP-T} & \text{LPIPS} \\ \hline \text{FLUX-LoRA(FP)} & W16A16 & 0.3564 & 0.6490 & 0.2501 & 0.8383 \\ \text{FLUX-Ours(MUL)} & W8A8 & 0.3150 & 0.6272 & 0.2348 & 0.8372 \\ \hline \end{array}

It can be seen that the performance drop of our IntLoRA compared to the original LoRA is acceptable, while achieving inference speedup. This preliminary result shows the potential generalization of the proposed method on other Diffusion model backbones.

Efficiency Comparison.

In Tab.3 of the paper, we give the forward time cost of MUL and SHIFT: 0.87s/img for MUL, and 0.84s/img for SHIFT. One can see that the SHIFT is relatively faster than the MUL. However, it is worth noting that the speedup of the log2 quantized model can be further accelerated by hardware-level optimizations. Since this work focuses primarily on the algorithmic level, we leave further log2 acceleration for future work.

Typos.

We will fix the types in the caption of Figures 4-6 in the revision. Thanks for your suggestion!

Responds to other Questions.

In weight merging of previous methods, i.e., $\mathbf{W'} = \mathcal{Q}(\mathbf{W) + AB}$, the $\mathcal{Q}(\mathbf{W})$ is the quantized INT8 weights, but $\mathbf{AB}$ is low-rank FP16 weights. This arithmetic inconsistency between FP16 and INT8 caused the $\mathcal{Q}(\mathbf{W})$ to have to upcast back to FP16 to allow addition with LoRA weights. As a result, the summed results $\mathbf{W'}$ is in FP16 type and needs PTQ to accelerate inference.

More discussions of Adaptation Quantization Separation (AQS).

The main motivation of the proposed AQS is to address the effect of zero-initialized weights of LoRA during quantization. Specifically, the zero-initialized weight leads to infinite results during quantization due to the divide-by-zero (please refer to Eq.1 in the original paper). In order to maintain the zero-initialized weights while achieving the quantization-friendly distribution, we propose the AQS, which decomposes the LoRA weights into a zero-initialized part that requires gradients (corresponding to the vanilla LoRA), and a non-zero part that does not require gradients (for ease of quantization) to facilitate quantization. In this way, AQS facilitates both model learning and quantization.

Correlation with Other Compression Methods, e.g., Knowledge Distillation.

Our approach focuses primarily on quantization to reduce the memory footprint during fine-tuning large models. However, since quantization and other compression methods are orthogonal technically, our approach can therefore potentially be combined with other methods such as distillation to achieve further computational cost reduction.

Specifically, we can first use the large pre-trained model as a teacher model and first obtain a small student model. Afterwards, we can use the proposed IntLoRA on the student model to further obtain quantized downstream task weights, thus realizing further speedup.

Since the knowledge distillation requires complete back-propagation of the model gradients, we cannot finish this process due to limited computational resources. However, given the promising performance of IntLoRA on existing small models (which can be viewed as distilled student models), combining IntLoRA and knowledge distillation is practical. We will give a more detailed discussion in the revision.

Responds to Other Questions.

• We assume you refer to the rank in the LoRA. Following the common practice, we determine the LoRA ranks through ablation experiments in Fig.9 in the Appendix. Performance generally improves as we increase the rank, but the rate of growth varies. To balance a tradeoff, we choose $r=4$ in all of our experiments.

• The motivation of VMC is to adjust the distribution shape of the adaptation term for effective log2 quantization. In detail, the log2 quantization requires that most of the values are distributed near zero to use as many log buckets as possible to reduce quantization error. As shown in Fig.8 in the Appendix, the proposed VMC can control the distribution shape to achieve sharp peaks and light tails, thus utilizing more buckets on the logarithmic scale to reduce the quantization error.

Why Introduce Many LLM Related Works?

Thanks for your comment! Although there is a lot of work on Diffusion quantization, very little work explores the adaptation of quantized diffusion, which is also recognized by Reviewer wfKN. Therefore, we introduce related methods in quantized LLM adaptation to ensure a comprehensive baseline comparison.

Clarification of the Claims.

Consistent with the title, this work mainly focuses on Diffusion models. The experiments on NLP tasks are preliminary explorations to answer potential concerns about the method generality. As suggested, we will make the relationship between Diffusion and LLM clearer to avoid any misleading or over-claims.

Efficiency of Different Stages.

In Tab.3, we give the training speed and memory usage during fine-tuning stage. Our IntLoRA is similar to that of QLoRA, but we can directly obtain the quantized merged weights.

For the pre-processing and post-processing stage, we give the time cost as follows.

\begin{array}{l|ccc} \hline \text{stage} & \text{pre-processing} & \text{fine-tuning} & \text{post-processing} \\ \hline \text{time} & 28.8s & 128.2s & 0.5s \\ \hline \end{array}

The pre-processing accounts for 18% total time. However, this costs can be shared across different tasks.

For weight merging, the latency can be neglected.

Justification of Quantized Adaptation & Why Dreambooth needs Quantization & Comparison with Text-Inversion.

Our quantized adaptation enjoys benefits in both tuning and inference. Specifically,

1. The low-bit adaptation reduce GPU usage during tuning. Although Dreambooth involves small samples, it still struggles when fine-tuning large models. For example, loading FP16 FLUX consumes over 40GB, let alone fine-tuning with Text-Inversion or LoRA. In contrast, our 4-bit quantized IntLoRA reduces the loading memory to <15GB, facilitating tuning on consumer-level GPUs.

2. The quantized downstream model boost inference efficiency. As shown in the Response to Reviewer 1Zzg, the INT GEMM is even more efficient than FP low-rank matmul, allowing for fast inference. As a comparison, the Text-Inversion has the same latency of the pre-trained large model.

As shown in Tab.1, our IntLoRA can achieve acceleration with negligible performance loss. We will include more discussion and related works in the revision!

Comparison with LQ-LoRA and GPTQ-LoRA.

Both LQ-LoRA and our IntLoRA share the idea of avoiding zero-initial low-rank weights. But they have several essential differences. 1) The formulation is different. LQ-LoRA uses the decomposition $\mathbf{W'} = \mathcal{Q}(\mathbf{W}) + \mathbf{AB}$ to achieve non-zero LoRA weights, our IntLoRA introduces the auxiliary matrix $\mathbf{R}$ and use $\mathbf{W'} = \mathcal{Q}(\mathbf{W + R) -R +AB}$ for this goal. 2) LQ-LoRA requires multi-iterations to search for the optimal approximation, while our IntLoRA does not. 3) We also compare LQ-LoRA on Dreambooth as follows. One can see that our IntLoRA achieves better performance.

\begin{array}{l|ccccc} \hline \text{methods} & \text{nbits} & \text{DINO} & \text{CLIP-I} & \text{CLIP-T} & \text{LPIPS} \\ \hline \text{LQ-LoRA} & W8A8 & 0.4056 & 0.6624 & 0.2824 & 0.8126 \\ \text{Ours-MUL} & W8A8 & 0.4498 & 0.6882 & 0.2858 & 0.8062 \\ \text{LQ-LoRA} & W4A8 & 0.4022 & 0.6797 & 0.2680 & 0.8198 \\ \text{Ours-MUL} & W4A8 & 0.4242 & 0.6913 & 0.2710 & 0.8181 \\ \hline \end{array}

As for GPTQ-LoRA, it appends LoRA weights on the GPTQ quantized models, which needs additional PTQ during inference deployment. Although GPTQ-LoRA is not accepted and not open-source, we are happy to add related discussion in the revision.

The Cost of R & Ablation on sigma_R.

• As stated in Line215, the $\mathbf{R}$ can be generated on-the-fly under the same random seed, and can be deleted once used. The performance in all tables has included the effects from $\mathbf{R}$.

• Since $\mathbf{R}$ is first normalized and then rescaled by $\alpha$, the ablation of $\alpha$ in Fig.7 can reflect the role of $\sigma_R$. One can see the performance is robust when $\alpha >1.4$. We adopt a fixed $\alpha = 1.5$, i.e., the same $\sigma_R$, in all experiments and the results are satisfactory.

Clarification on Fig.1 & Dtype of Adaptation-Term.

Similar to common QAT practice, the weights and activations (and Adaptation term) during training are "simulatively quantizated" with FP16 dtype, for accurate gradient backpropagation. However, the inference is strictly in INT dtype (or 2^X format for SHIFT). Fig.1 is the post-tuning weight merging stage, so it is correct since both W and A are INT. We will clarify this in revision.

Solution to Divide-by-Zero.

Because $\mathbf{W}_{round}-z$ is ensured to be an integer, we replace the 0 with 1 and then zero mask the division result at corresponding slots.

The significance of weight merging

Good comments! In fact, the efficiency of INT-type matrix multiplication is extremely efficient in the highly optimized GEMM on modern GPUs, and we demonstrate below the INT8 matmul is even faster than that of FP32 low-rank matrix multiplication.

Given a weight matrix $\mathbf{W} \in \mathbb{R}^{M\times N}$, activation $\mathbf{X} \in \mathbb{R}^{N \times L}$ and the low-rank decomposition $\mathbf{W = AB}$ ( where $\mathbf{A} \in \mathbb{R}^{N \times r},\mathbf{B} \in \mathbb{R}^{r \times M}$), we adopt the shapes from SD-1.5, i.e., $M = N = 320, L = 4096, r=4$. We use INT8 GEMM CUDA kernel to calculate $\mathcal{Q}(\mathbf{W})\mathcal{Q}(\mathbf{X})$ ($\mathcal{Q}$ denotes the quantization operator), and use the common FP32 torch.matmul in Pytorch to compute the $\mathbf{ABX}$. The experimental results on NVIDIA A5000 GPU are as follows:

\\begin{array}{l|ccc} \\hline \text{setups} & \text{runtime} & \text{GPU} \text{costs} & \text{FLOPs} \\\\ \\hline \text{INT8 GEMM} & \text{48.71 ms }& \text{1.34MB} & \text{0.8388G} \\\\ \text{FP32 matmul} & \text{52.16 ms }& \text{5.01MB} & \text{0.0209G} \\\\ \\hline \\end{array}

Although full-rank matrix multiplication has 40x higher FLOPs than its low-rank counterpart, the INT matrix multiplication shows even faster speeds and less GPU footprints due to quantized INT8 type on integer-optimized GEMM kernels.

From the above analysis, the low-rank FP32 matmul is not negligible than the INT8 pre-trained weights. Therefore, it makes sense to apply the proposed IntLoRA to directly obtain the quantized merged weights.