QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models

We thank the reviewer for the comments and questions.

Q1. Parameter offset in experiments: The proposed method incorporates group-wise/sub-channel quantization, which includes an additional number of parameters for scales. Also, the proposed QA-LoRA reduces the size of low-rank matrices. However, these parameter offsets are not reflected in the results, which could be misleading to the audiences. It would be more informative to add the actual model size (or estimated) in MB/GB for each of the models.

A1. Thanks for the suggestion. First, we report the sizes of the final models of QLoRA and QA-LoRA in the following table. Please note that there are two ways for post-processing in QLoRA, i.e., the unmerged ($\tilde{\mathbf{W}}$ and $s\cdot\mathbf{A}\mathbf{B}$ are stored individually, which saves memory but the inference is slow) and merged ($s\cdot\mathbf{A}\mathbf{B}$ is added to $\tilde{\mathbf{W}}$, which is faster in inference but requires large memory because the matrix must be stored in FP16). QA-LoRA enjoys both low memory usage and a fast inference speed. A side note: In 33B and 65B models, setting $L=32$ in QA-LoRA results in slightly larger model sizes compared to QLoRA, but one can set $L=128$ which causes a negligible accuracy drop. Note that the final model size of QA-LoRA is exactly the size of $\mathbf{W}'$ (or equivalently, $\tilde{\mathbf{W}}$) because $s\cdot\mathbf{A}\mathbf{B}$ is merged into $\tilde{\mathbf{W}}$ after adaptation. Take the 7B model with $L=32$ as an example. The baseline, the unmerged version of QLoRA, is sized 4.6G, in which $\tilde{\mathbf{W}}$ is sized 4.0G and $\mathbf{A}$ and $\mathbf{B}$ combined is sized 0.6G. QA-LoRA increases the first amount to 4.3G and eliminates the second amount. In the revised paper, we have added the following tables to Appendix C.

Table. The model size (GB) of QLoRA and QA-LoRA with respect to different options. The QLoRA models used NF4 and FP16 numerics.

Q2. In the ablation study, only group size is examined. It would be worthwhile to experiment with the D_int as well, as it is also part of the tradeoff between model size and accuracy. It would be interesting to see what is the lowest D_int in this setup, compared to vanilla LoRA.

A2. Good suggestion! We follow QLoRA to set $D_\mathrm{int}=64$. During the rebuttal, we diagnose the performance with respect to $D_\mathrm{int}$ in the following table. We find that the MMLU accuracy is not largely impacted by the value of $D_\mathrm{int}$ unless it is too small (the same conclusion as in the QLoRA paper). We agree that discovering the smallest $D_\mathrm{int}$ value is interesting, and we empirically find that the smallest $D_\mathrm{int}$ that maintains the stability of QA-LoRA is $4$. In the revised paper, we have added the table to Appendix B and the analysis to the main article.

Table. The MMLU accuracy (%) of $D_\mathrm{int}$ measured on INT4 and INT2 quantization upon LLaMA-7B on the Alpaca dataset.

We thank the reviewer for the comments and questions.

Q1. QA-LoRA introduce a hyper-parameter ($L$: group size). This requires additional optimization and it is unclear if it can be selected without tuning.

A1. Thanks for the question. We first analyze the impact of $L$. A smaller $L$ results in (1) a higher degree of freedom (which often leads to higher fine-tuning accuracy), (2) a larger memory to store the quantized weight matrix, $\tilde{\mathbf{W}}$, and (3) an increasing risk of over-fitting if $L$ is too small. Therefore, a good strategy is to set $L$ to be a relatively small (but not too small) integer, e.g., $L=32$ as a good practice as shown in Table 5. We cannot guarantee that no hyper-parameter tuning is needed, but there is a clear guideline to do this. For example, one can try $L=16$, if the GPU memory allows, to achieve higher accuracy.

Q2. I wonder if a larger model where the need for this technique is crucial, e.g., 30B-60B, could be discussed.

A2. Nice suggestion! In the original version, we provide experiments on the 33B and 65B models of LLaMA in Table 1, and QA-LoRA achieves improvement over the baseline, QLoRA with GPTQ. Upon these results, we discuss the need for QA-LoRA in large models in the following aspects. (1) QA-LoRA reduces the computational costs of fine-tuning large models, e.g., using QA-LoRA, only 1 and 2 V100 GPUs are needed for fine-tuning the 33B and 65B models. (2) When larger models are used, there can be increasing needs for low-bit (e.g., INT3 and INT2) quantization, especially when the large models are to be deployed to edge devices. QA-LoRA shows significant advantages in such scenarios. In the revised paper, the above contents are added to Section 4.2 where the experimental results are discussed.

Q3. Figure 3 legend should have QA-LoRA instead of A-LoRA.

A3. Sorry for our carelessness. It has been fixed in the revised version.

=====continuing from Part 1=====

Q4. In Table 2, you indicate that the number of parameters in the method is lesser than QLoRA — almost by 2x. Why is this the case? This seems like an unfair comparison to QLoRA. What is the time taken in hours for fine-tuning with a similar number of parameters?

A4. The reason behind the fewer amounts of parameters, compared to QLoRA, lies in the reduction of the dimensionality of $\mathbf{A}$. Compared to LoRA and QLoRA where $\mathbf{A}$ has $D_\mathrm{in}\times D_\mathrm{int}$ parameters, QA-LoRA reduces the number to $L\times D_\mathrm{int}$ where $L$ is the group size and $L\ll D_\mathrm{in}$. This reduces the number of parameters in QA-LoRA by around $1/2$ (originally, $\mathbf{A}$ and $\mathbf{B}$ have similar numbers of parameters). We would like to stress that this is a good feature of QA-LoRA which allows it to achieve higher fine-tuning accuracy with fewer parameters. Regarding the fine-tuning time, it is not largely impacted by the parameters because the amount of LoRA parameters is much smaller than that of the LLM itself (e.g., 89M or 160M vs. 7B). To verify this point, we double $D_\mathrm{int}$ which also doubles the number of parameters, surpassing that of QLoRA, but QA-LoRA is still much faster than QLoRA (see the table below). Essentially, QA-LoRA is faster in training because it uses INT4 computation while QLoRA uses NF4 computation which is not well optimized by CUDA. In the revised version, we make this point clear by adding the above contents to Section 4.2 (the paragraph starting with "the efficiency of QA-LoRA").

Table. A comparison of the number of parameters and training time.

Q5. Section 3.3 explains the method and the reasoning behind why the rank degenerates to 1. But the explanation is not comprehensive.

A5. First of all, please note that the final step of QA-LoRA is to fuse $\tilde{\mathbf{W}}$ (the quantized matrix during training) and $s\cdot\mathbf{A}\mathbf{B}$ (the adaptation weights) into a new matrix $\mathbf{W}'=\tilde{\mathbf{W}}+s\cdot\mathbf{A}\mathbf{B}$. For the efficiency of inference, we hope that $\mathbf{W}'$ is still in low-bit quantization. This means that all values in $\tilde{\mathbf{W}}$ and $\mathbf{W}'$ must come from an equidistant discrete set (see Footnote 1). As an example, let a column of $\tilde{\mathbf{W}}$ contain the numerical values from the set of $\{0.3,1.5,2.7,3.9\}$, where we can use INT2 quantization ($\alpha=1.2$, $\beta=0.3$). Now, it is added by the corresponding column in $s\cdot\mathbf{A}\mathbf{B}$ (both $\mathbf{A}$ and $\mathbf{B}$ contain continuous numerical values), and we require that the summation is still INT2-quantizable. To guarantee this in a continuous optimization procedure, the only known way is to constrain the added values to be constant, which derives that all row vectors of $\mathbf{A}$ are identical, and hence $\mathrm{rank}(\mathbf{A})=1$. Actually, we can choose not to say $\mathrm{rank}(\mathbf{A})=1$ but simply say that the flexibility of adaptation is largely constrained because the row vectors of $\mathbf{A}$ must be identical. We hope that we have made the explanation comprehensive. Of course, we are open to further discussions. Thanks.

We thank the reviewer for the comments and questions.

Q0. Reasoning behind the method - Why should all the c_ij as defined in the paper be equal is not clear — which is the main motivation for the group-wise quantization.

A0. Please refer to our response to Q5.

Q1. In Algorithm 1, the function merge_with_quantization is defined but never used.

A1. Sorry for misleading. Due to the space limit, we did not add the main function in the paper, which calls qalora_training (which calls qalora_forward) and merge_with_quantization. The merge_with_quantization function is called after the entire QA-LoRA training procedure for merging the LoRA weights $s\cdot\mathbf{A}\mathbf{B}$ into the quantized weight matrix $\tilde{\mathbf{W}}$. It is done by updating the $\beta$ parameter into $\beta_\mathrm{new}$. In the revised paper, we have added an explanation in the main article, after the reference to Algorithm 1.

Q2. What is the degree of freedom of quantization and adaptation - These seem to be new terms that are added in this paper and not used in the literature. These have to be defined, before claiming that they are increased or managed by the proposed method.

A2. Thanks for the question. Literally, the degree of freedom (DoF) means how many parameters (or pairs of parameters) can be used for quantization or adaptation. We explain our insight as follows and, to avoid confusion, we replaced all "the degree of freedom" with "the number of parameters" in the revised paper. In QLoRA, each column of $\mathbf{W}$ is quantized using the same pair of $\alpha$ and $\beta$ parameters, which we say the DoF (number of parameters) of the quantization part is $1$; meanwhile, the matrix $\mathbf{A}$ is sized $D_\mathrm{in}\times D_\mathrm{int}$, implying that each of the $D_\mathrm{in}$ entries in the column of $\tilde{\mathbf{W}}$ is multiplied by an individual parameter, which we say the DoF (number of parameters) of the adaptation part is $D_\mathrm{in}$. There is clearly an imbalance here, which causes two issues: (1) the over-high quantization error harms the fine-tuning accuracy, and (2) the adaptation weights cannot be merged to the quantized matrix, $\tilde{\mathbf{W}}$. This motivates us to (1) increase the number of parameters of the quantization part from $1$ to $D_\mathrm{in}/L$ using group-wise quantization, where $L$ is the group size, and (2) decrease the number of parameters of the adaptation part by squeezing the input dimensionality of $\mathbf{A}$ from $D_\mathrm{in}$ to $L$. This is why QA-LoRA improves the accuracy and naturally allows the adaptation weights to be merged into the quantized matrix.

Q3. Page 6 is just results - Page 6 is just results without much to interpret. These are a bunch of numbers. Please consider presenting this table in a better manner. Can this be presented as a graph for readers? Just numbers are hard to read.

A3. Thanks for the suggestion. We followed QLoRA and other recent works to report these exact numbers to ease the comparison against existing papers. Actually, the main results of Table 6 (5-shot MMLU on the Alpaca dataset) have been summarized in Figure 1. In the revised paper, we plot other results (0-shot MMLU and the FLAN v2 dataset) as similar figures and add them to Appendix B.

=====to be continued in Part 2=====

Thanks for the comment. We hope that the question and our response are helpful for you and the referees to further understand the details of our paper.

You are right. We used the AutoGPTQ library and QLoRA used the bitsandbytes library. This is because QLoRA was built upon the NF4 format which is not supported by any library except for bitsandbytes (created by the same authors); meanwhile, bitsandbytes does not support INT4 (as well as INT3 and INT2). That is why we cannot compare the two algorithms in exactly the same setting. In the inference stage, the speed of QA-LoRA is about $2\times$ of that of QLoRA (QLoRA is ran on bitsandbytes and QA-LoRA on AutoGPTQ).

Regarding the advantage of training costs, we agree that QA-LoRA has a similar (slightly smaller, due to the reduced size of $\mathbf{A}$) number of operations as QLoRA. QA-LoRA's advantage in training costs mainly comes from the faster speed in executing INT4 (integer) operations compared to that in executing NF4 (floating point number) operations. This is one of the main advantages of QA-LoRA; besides, other advantages of QA-LoRA include:

• QA-LoRA allows merging $\tilde{\mathbf{W}}$ and $s\cdot\mathbf{A}\mathbf{B}$ into one quantized matrix, making the inference easier and more efficient.
• QA-LoRA achieved higher accuracy in the MMLU tasks, spanning over different backbones and datasets.

Again, we appreciate the comments which allow us to clarify this point during the rebuttal period. Thanks.