IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models

[Q1: Discussion on the training costs]

The training costs are not reported in the paper.

As suggested, we compare on the training speed and GPU usage on the SD-1.5 based Dreambooth finetuning task. We use the NVIDIA 3090 GPU for testing, and the results are shown as follows.

It can be seen that our IntloRA exhibits slightly larger training costs than LoRA-FP. This makes sense due to the fact that QAT typically consumes more cost than the common training counterpart. However, our IntLoRA can achieve accelerated inference benefiting from the quantization despite its slightly larger training cost than LoRA-FP. Moreover, comparing with QLoRA, which also uses QAT-like training, our IntloRA achieves similar training complexity, but our approach does not require additional PTQ after tuning, facilitating user-customized fast and low-cost deployment.

[Q2: Contribution on INT type adapter]

In section 5.2, the authors mentioned that “IntLoRA is the only one that achieves INT type adaptation weights”, which is not true since EfficientDM also uses quantized LoRA weights.

We would like to clarify that EfficientDM actually implements quantized merged weights, i.e., $\mathcal{Q}(\mathbf{W+AB})$, rather than our IntLoRA-like quantized adapter weights. This difference is also well recognized by Reviewer wQ3r.

Quantized adapter weights are more challenging compared to quantized merged weights. This is because quantized merged weights only need to quantize the summation result of $\mathbf{W}$(FP16) and $\mathbf{AB}$(FP16), whereas quantized adapter weights require the quantization parameters of the adapters match those of the pre-trained weights for subsequent weight merging. Our IntloRA employs techniques including AQS, MLA and VMC to achieve the quantized adapter weights, thus facilitating seamless quantized weight merge without additional PTQ.

[Q1: Why the all-zero distribution is not quantization-friendly]

The text should elucidate why such a distribution necessitates a distinct quantizer design at the beginning of tuning. Providing a detailed example or analytical explanation on gradients would greatly enhance understanding, highlighting the specific challenges and potential impacts on the model’s performance during quantization.

Thank you for the comment. The all-zero distribution is difficult to quantize due to its limited value ranges. In detail, given the scaling factor $s = [\max(X)-\min(X)] / (2^b-1)$, this all-zero distribution leads to $min(X) = max(X)$, i.e. $s=0$. Considering that the quantization of $X$ in Eq.1 appears the division by $s$, this all-zero distribution can lead to large quantization error due to the infinity value.

We also attempt to explain this using the gradient analysis as you suggested. Specifically, since the AQS is proposed to avoid this all-zero distribution, we thus remove AQS from IntLoRA to obtain the zero-initilized LoRA weights and we try to train it to observe the gradients and performance. The results are as bellow.

We observe that the training collapses at the beginning with a nan gradients when we remove the AQS to train a zero-initilized variant. This ablation further validates the necessity of our AQS.

[Q2: The Determination of $r$ and $\alpha$]

In the description of the VMC, the authors introduce a scalar $\alpha$ used as an exponent of $r$ to finely adjust the search for the optimal auxiliary matrix. However, the method for conducting this search remains unclear. It would be beneficial for the authors to specify the objective function used in this search process. Clarifying how $\alpha$ and $r$ are optimized, including the criteria and algorithms employed, would provide a more comprehensive understanding of the method’s effectiveness and implementation.

In the proposed VMC, we use equation $r = \sigma_\mathbf{W}/\sigma_\mathbf{R}$ to determine the $r$, as illustrated in Line273 in the submission. After that, we use grid search from 0 to 2 with a step size 0.1 to determine the optimal $\alpha$. The objective function of the grid search is task-oriented, i.e., we use the task metrics to select the optimal $\alpha$, and the performance with varying $\alpha$ is shown in Fig.7.

[Q3: Integerate IntLoRA to Network-wide reconstruction]

The proposed method, IntLoRA, demonstrates substantial potential to enhance the performance of quantized models beyond simple adaptation. Integrating IntLoRA’s innovative techniques, such as integer low-rank parameters and network-wide reconstruction calibration, not only maintains but also significantly boosts the efficacy of models operating under quantization constraints.

As suggested, we apply our IntLoRA to the diffusion model quantization, similar to the setup of EfficientDM. The experimental results on W4A4 quantized LDM-4 model on the ImageNet 256x256 image generation task are given in the global author response. The results show that our IntLoRA can also be applied to the network quantization, demonstrating the generality of the proposed method.

[Q4: Impacts from the auxiliary matrix]

The authors introduce the use of an auxiliary matrix in the AQS. It would be helpful to clarify whether the introduction of this auxiliary matrix introduces additional quantization difficulty like outliers to the pre-trained model weights.

In fact, the proposed VMC can effectively mitigate the outliers from the auxiliary matrix $\mathbf{R}$. Specifically, we use the variance ratio $r = \sigma_\mathbf{W}/\sigma_\mathbf{R}$ in the VMC to rescale the original sampled $\mathbf{R}$, which ensures that the value range of $\mathbf{R}$ is smaller than the one of $\mathbf{W}$, thus avoiding the introduction of additional outliers.

To validate it, we also give the distribution of the $\mathbf{W}$, as well as the distribution of VMC-rescaled R in Fig.13 Appendix-G in the revised PDF. It can be seen that the range of $\mathbf{R}$ is well covered by $\mathbf{W}$, thus ensuring a minimal quantization error of $\mathcal{Q}(\mathbf{W-R})$.

[Q5: Comparison with EfficientDM]

The paper overlooks an important baseline in its experimental setup or discussion. The study “EfficientDM” by He et al., 2023, which similarly employs low-rank parameters for fine-tuning pre-trained models, is notably absent. Like IntLoRA, EfficientDM also allows for the merging of low-rank parameters into pre-trained model weights post-training. Including this baseline could enhance the comparative analysis, providing a clearer benchmark of IntLoRA’s performance against existing methodologies with similar strategies.

We provide a comprehensive comparison with EfficientDM in the global author rebuttal [G2], including using the quantized merged weights of EfficientDM for downstream fine-tuning tasks, as well as using our integer adapter for diffusion model quantization.

From these experimental results, one can see that our IntLoRA can achieve better performance and even outperforms EffcientDM on W4A4 network quantization. It is worth noting that this work mainly focuses on downstream adaptation considering the benefits from the quantized adapter and seamless quantized merging. We will give a detailed discussion with EffciientDM in the revised version.

[Q6: Strategy to overcome value overflow during merging]

The authors merge integer low-rank parameters with quantized pre-trained model weights after training, a process that might lead to value overflow. It would be beneficial for the paper to address how this potential issue is managed. Specifically, detailing the mechanisms or techniques employed to prevent or mitigate overflow—such as scaling factors, normalization methods, or safeguards within the merging algorithm—would provide a clearer understanding of the reliability of the proposed method in handling data integrity during integration.

First, the proposed IntLoRA-SHIFT, which directly bit-shifts the INT4/INT8 weights as shown in Eq.9, does not face the numerical overflow problem, since the 1/2^N in the equation can act as a scaling factor. Therefore, we mainly discuss the machinsm to overcome value-overflow with IntLoRA-MUL of Eq.7. In detail, we can accumulate intermediate multiplication values in a higher precision and then we can scale and quantize the result back to INT4. For instance, during the multiplication between INT4 low-rank parameters and pre-trained weights, we can use INT32 to hold the intermediate result. After that, we quantize the INT32 result as the product of the FP32 $s'$ and the INT4 merged weight. Finally, the FP32 $s'$ is multiplied with the original FP32 $s$ to obtain the final scaling factor and the INT4 merged weights.

Dear Reviewer SB93,

Thank you for taking the time and effort to review our paper. We have carefully addressed your valuable comments in detail. As the discussion phase is coming to an end soon, we kindly invite any further comments or suggestions you may have. If we have addressed your concerns, please kindly consider upgrading the score.

We sincerely appreciate your efforts, which have significantly contributed to improving our manuscript.

Best regards, The Authors

[Q3: Discussion on training efficiency]

The proposed PEFT method does not reduce training memory usage compared to existing PEFT methods (e.g., QLoRA). Furthermore, quantization-aware training of adapters may slow down training and even increase memory usage. The paper should discuss the training speed and memory usage.

The core goal of existing PEFT methods is to reduce the training cost, yet neglecting subsequent inference deployment. As recognized by Reviewer wQ3r and SB93, our method is both training and inference efficient. It is unfair to compare IntloRA to exiting PEFT methods only on the training end, since IntLoRA not only achieve QLoRA-like training efficiency, but also enables seamless quantized inference without additional PTQ.

As suggested, we compare on the training speed and GPU usage on the SD-1.5 based Dreambooth fine-tuning task. We use the NVIDIA 3090 GPU for testing, and the results are shown as follows.

It can be seen that our IntloRA exhibits slightly larger training costs than LoRA-FP. This makes sense due to the fact that QAT typically consumes more cost than the common training counterpart. However, the drawback of LoRA-FP is that it is difficult to speed up in the inference phase. Moreover, comparing with QLoRA, our IntloRA achieves similar training complexity, but our approach does not require additional PTQ, facilitating user-customized local deployment.

[Q4: Storage requirements for $\mathbf{R}$]

Storing $\mathbf{R}$ after training the adapters would require significant storage, which may limit the method's practicality. If $\mathbf{R}$ is not stored, the authors should explain how it is obtained during inference. Also, note that $\mathbf{R}$ should derive from $\mathbf{W}_{round}$ rather than $\mathbf{W}$, as only quantized weights are accessible during PEFT.

It should be noted that $\mathbf{R}$ is a distribution and is task-shared. Thus, given the distribution parameters and a fixed random seed, we can reproduce $\mathbf{R}$ online, meaning that $\mathbf{R}$ does not need to be stored.

As for the derivation of $\mathbf{R}$, since we only need the max-value of $\mathbf{W}$ as mentioned in Line 409 of paper, we can thus approximate $max(\mathbf{W})$ as $s \cdot (max(\mathbf{W_{round}})-z)$, i.e., only $\mathbf{W}_{round}$ is needed to derive the VMC controlled $\mathbf{R}$.

[Q5: About the writing]

We will revise the paper based on the comments and update the PDF soon. Thank you for the detailed suggestions!

[Q1: Clarification of benefits from weight-only quantization]

The paper exclusively focuses on weight-only quantization, where activations remain at 16-bit precision. Since diffusion models are primarily compute-bound, quantizing only the weights does not yield speedups during inference.

Thanks for rising this concern. We would like to point out that the weight-only quantization of IntloRA can reduce GPU memory usage during fine-tuning, which is meaningful under the user customization senarios. Specifically, it is chanlleging for LoRA-FP tuning to load large diffusion models, e.g., just loading FLUX.1-dev[1] weights can cost 23.8GB GPU memory, on consumer-level GPUs. In contrast, our IntloRA can directly work on the quantized model, which reduces the loading GPU cost and facilitates diffusion model fine-tuning for user customization.

[1] https://huggingface.co/black-forest-labs/FLUX.1-dev

[Q2: IntLoRA with Activation Quantization]

Additionally, in a weight-only quantization setting, users can choose not to fuse adapters, thereby avoiding re-quantization. As a result, the primary benefit of IntLoRA is just a reduction in adapter size compared to conventional PEFT methods—a relatively minor contribution, given that LoRA adapters are already small.

As suggested, we further supplement the activation quantization of IntLoRA. In detail, we employ the simple per-tensor Uniform Affine Quantization on the activation, without using other complicated tricks, and find it could work well on up to W4A8. The results of Dreambooth fine-tuning are as follows, in which we use the average results of 15 text-subject pairs for evaluation.

It can be seen that our IntLoRA maintains superior performance under activation quantization settings. Importantly, the addition of activation quantization allows our IntLoRA to achieve practical inference speed-up.

As for the W4A4 setup, we observe both our IntLoRA and all other baselines fail, which suggests additional techniques are needed for lower activation quantization bits. Considering that we mainly aim to address the challenge of arithmetic inconsistency during weight merge, we leave further 4-bit activation quantization for future work. Finally, we can also apply our IntLoRA to Diffusion Model Quantization, and even works well on the W4A4 setup. Please refer to the global author rebuttal for related experimental results.

[Q6: Reply to other questions]

6-1. Justification for $\mathbf{AB}$ being orders of magnitude smaller than $\mathbf{R}$

We give the figure of the distribution of the learned $\mathbf{AB}$ and $\mathbf{R}$ in Fig.13 Appendix-H in the revised PDF. Since $\mathbf{AB}$ in LoRA is zero- initialized, it tends to be distributed around zero with learned small values aiming to not disturb the pre-training weights too much.

6-2. What is the purpose of a zero-mean adaptation term, and why it is better?

We prefer zero-mean adaptation term in order to facilitate subsequent log2 quantization on the adaptation term. This is due to the fact that most log2 buckets are around zero, e.g., $2^{-12}, 2^{-13}$, and thus a zero-mean adaptation term with the VMC controlled variance can force the values to near the zero, thus reducing log2 quantization error. We also provide the counter-example in Fig.6-Right, where the distribution of non-zero means results in very few log2 buckets being used, eventually leading to this setup cannot work.

6-3. Table 4: Why does the storage reduction exceed 8× (e.g., 9.8/1.2>8 and 0.34/0.04>8)?

This is due to the value round for presentation. For example, the original model is saved in FP32, and for 4-bit quantization settings, the reported results are obtained from $9.8/8 = 1.225 \approx 1.2$, and $0.34/8 = 0.0425 \approx 0.04$. We will make the values more precise in the revised version.

6-4. Figure 6: Could you clarify the difference between VMC $\sigma_{R}$ and the appropriate $\sigma_{R}$ How can one observe the large quantization error from the second image in the second row of the figure?

The VMC $\sigma_\mathbf{R}$ actually means the appropriate $\sigma_\mathbf{R}$, i.e., they convey the same concepts, and we will unify the term in the revision.

The second image in the second row in Fig.6-Left is the reconstruction results of its left blue counterpart using the reconstruction equation: $W \approx \mathcal{Q}(W-\mathbf{R}) + \mathbf{R}$. The difference between these two blue KDE curves indicates large reconstruction errors due to a too large $\sigma_\mathbf{R}$.

Dear Reviewer qTL5,

Thank you for taking the time and effort to review our paper. We have carefully addressed your valuable comments in detail. As the discussion phase is coming to an end soon, we kindly invite any further comments or suggestions you may have. If we have addressed your concerns, please kindly consider upgrading the score.

We sincerely appreciate your efforts, which have significantly contributed to improving our manuscript.

Best regards, The Authors

[Q1: Comparison with EfficientDM]

Recent works have worked out quantized diffusion models using LoRA. Different from this paper, EfficientDM focuses on quantized merged weights at inference stage. Can you compare with it?

Thanks for your comment, we provide a detailed comparison with EfficientDM in the global rebuttal [G2].

In short, our IntloRA is more suitable for accelerated downstream fine-tuning compared to EfficientDM. In addition, our approach can also potentially be used for diffusion model quantization and have shown promising results.

[Q2: The significance of accelerated training]

I am doubting the importance of the speed-up for the training stage as the pretrained weights is fixed, only small adapters are updated. Can you show how much speed-up as compared to full-precision? or at least theoretically show improvement?

Thank you for raising this concern. We would like to point out that LoRA only reduces the GPU memory of gradient and optimizer states during training, and cannot benefit the model weights. Considering that the diffusion models are getting bigger, e.g., just importing FLUX.1-dev to cuda can take up 23.8GB, it makes sense to reduce the size of the weights during training.

Our IntloRA only needs the quantized pre-training weights during training, further facilitates user-customized diffusion fine-tuning on consumer-level GPUs. Theoretically, taking SDXL as an example, full fine-tuning SDXL consumes 59GB GPU memory, of which 13.5GB comes from the weights. Using FP32 LoRA tuning consumes 28.38GB, indicating 30.62GB reduced gradients and optimizer states. However, this LoRA tuning still maintains the weights in FP32. Theoretically, our IntLoRA can further reduce this 13.5GB FP32 weight to 3.375GB with the INT8 quantization.

Finally, we would like to remind that efficient training is only one of the benefits of our IntLoRA, whose another advantage is to obtain downstream quantized models without additional PTQ for efficient inference.

[Q3: Results on NLP tasks]

From my understanding, the proposed methods also can be used in NLP tasks. As the baselines have results on NLP tasks in their papers, why not verify the proposed methods on NLP tasks? Could you show some experimental results on NLP tasks?

We understand your concerns, that is why we conduct experiments on mathQA task with Llama3-8B model in Tab.6 in the Appendix-A, and we also give this result as follows. It can be seen that our W8 IntloRA fine-tuned model outperforms other quantization baselines, indicating the generalizability of the proposed method for other tasks.

[Q4: Reliability of the results]

For diffusion models, seems it have standard/common verification on some tasks, please check EfficientDM paper. As all baselines are implemented by authors, I just doubt the reliability because of hyper-parameter tuning. Maybe it is better compare with SOTA results in paper.

As suggested, we replace the EfficientDM-Layer(line 385) in quant_layer.py [5], in EfficientDM github repo with our IntLoRA-Layer and keep the others intact to validate on the diffision model quantization task. The results are shown in the global rebuttal [G2], and it can be seen our IntLoRA can achieve better performance. As discussed in the global author rebuttal [G2], our IntLoRA is more suitable for applying to downstream adaptation. To this end, in the paper, we follow the evaluation protocol in diffusion model fine-tuning works, including OFT[1], COFT[2], as our core experimental task setup.

As for the reliability of the compared baselines, the implemented methods for comaprasion also follows their official code: QLoRA[3], QALoRA[4], IR-QLoRA[5], and we keep all other hyper-parameters the same and only change the module class of Quantized-Linear-Layer with different methods.

Finally, in the attached suppl-zip, we also provide the code of our IntLoRA on Dreanbooth fine-tuning task. And we will open-souce all related code upon accptance.

[1] Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS23. Code: https://github.com/Zeju1997/oft

[2]Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR24. Code:https://github.com/wy1iu/butterfly-oft

[3]QLoRA: https://github.com/artidoro/qlora

[4]QALoRA: https://github.com/yuhuixu1993/qa-lora/blob/main/peft_utils.py

[5]IR-QLoRA: https://github.com/htqin/IR-QLoRA/blob/main/integer%20quantizer/quantizer_icq.py

[6]EfficentDM: https://github.com/ThisisBillhe/EfficientDM/blob/main/quant_scripts/quant_layer.py

Dear Reviewer wQ3r,

Thank you for taking the time and effort to review our paper. We have carefully addressed your valuable comments in detail. As the discussion phase is coming to an end soon, we kindly invite any further comments or suggestions you may have. If we have addressed your concerns, please kindly consider upgrading the score.

We sincerely appreciate your efforts, which have significantly contributed to improving our manuscript.

Best regards, The Authors

Dear Reviewer wQ3r,

We notice it has been 6 days since our Author Rebuttal to your raised concerns. We understand your tight schedule, but given that the discussion phase is about to end, we sincerely hope that you can update your assessment if our response addresses your questions. If you have any other questions, we will actively discuss with you as well.

Thank you for taking the time and effort to review our paper.

Best regards, The Authors