LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

Thanks for the comments --- please see our responses below.

Comparison with methods that use the data $X$ to quantize

Thanks for the question! We actually did implement and compare against a baseline that uses OPTQ/GPTQ to quantize first before learning low-rank updates. In particular, we use GPTQ to first quantize the model and then learn low-rank updates. We called this "GPTQ-LoRA" baseline "LREC" in the paper following this paper (https://arxiv.org/pdf/2306.08162.pdf). We find that GPTQ-LoRA underperforms both QLoRA and LQ-LoRA (Table 1).

(We now realize that this naming caused some confusion. Thanks for pointing that out --- we will clarify this in the paper.)

Additional Post Training Quantization Experiments

In addition to the GPTQ-LoRA experiments already in the paper, we performed additional experiments using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset. Specifically, we train the LQ-LoRA (Fisher, 2.75-bit, 64-rank) for one epoch on two C4 partitions and WikiText-2. We then further quantize the LoRA components themselves to 8-bit (NF8). Please see the Table below for the results, where the numbers other than LQ-LoRA are from OmniQuant [1]. Note that we use the term "effective bits" to denote bits per parameter that treat the LoRA components as "overheads" for comparing with methods without LoRA; LoRA components (with NF-8 quantization) amount to about 0.1-0.2 extra bits per parameter.

[1] https://arxiv.org/abs/2308.13137

These results indicate that LQ-LoRA can also be used as a state-of-the-art PTQ method. Given the promise of LQ-LoRA as a potential alternative to PTQ, we will include this comparison against PTQ results in the next iteration of the paper.

NF-based vs INT-based Quantization

You are correct in that the techniques in this paper are not specific to NF. QLoRA paper demonstrated the superior quality of such a scheme compared to classical integer quantization. Our preliminary experiments confirmed this, and hence, we chose NF.

We conducted the following experiment to be more concrete about NF vs INT. We ran LQ (i.e., Section 3.1) on LLaMA-2 7B's dense parameters with rank=64 and NF3/INT3/NF4/INT4 quantization configuration (i.e., b0=3/4, b1=8, b2=fp32, B0=64, B1=256), and measure the reconstruction errors (i.e., $||\mathbf{W} - (\mathbf{Q} + \mathbf{L}_1 \mathbf{L}_2) ||_2^2$). The table below shows that NF indeed performed better.

Thank you for your detailed response.

However, my concerns are not still completely addressed. First of all, the accuracy of LREC $4$-bit on MMLU almost matches that of LQ-LoRA, which seem to make the motivation of LQ-LoRA weaker. Furthermore, the authors compare their method with GPTQ-LoRA only. As GPTQ is a quantization method based on a layer-wise optimization, if a quantization method based on a block-wise optimization such as BRECQ [1] and FlexRound [2] is employed, "BRECQ-LoRA" or "FlexRound-LoRA" could surpass LQ-LoRA. Then, in my view, the motivation of LQ-LoRA would become very weak. To strengthen the motivation of LQ-LoRA, it would be necessary to compare LQ-LoRA with "BRECQ-LoRA" or "FlexRound-LoRA".

[1] BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction, ICLR 2021

[2] FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization, ICML 2023

Thank you for the discussion! Please see our responses below.

1. The accuracy of LREC-4bit on MMLU almost matches that of LQ-LoRA

For both LLaMA-2 7B and 70B models, LQ-LoRA (Fisher) with 3.25 bits can match the performance of LREC/GPTQ-LoRA 4-bits (i.e., 4.156-bits). Similarly, LQ-LoRA (Fisher) with 2.5/2.75 bits can match the performance of LREC/GPTQ-LoRA 3-bits (i.e., 3.148-bits). Please see the table below.

$^\dagger$Note that we additionally filled in the MMLU 70B performance for GPTQ-LoRA 3-bit; this number was absent from the initial submission. Based on the above, we conclude that LQ-LoRA can meaningfully improve upon GPTQ-LoRA and QLoRA.

2. The authors compare their method with GPTQ-LoRA only

Besides GPTQ-LoRA, we also compared our method QLoRA and different variants of LQ-LoRA. In addition, our latest PTQ experiments included comparisons with RTN, GPTQ, AWQ, and OmniQuant. For details, please see "Additional Post Training Quantization Experiments" in our earlier response.

3. BRECQ and FlexRound

Thanks for the suggestions! We think applying other quantization techniques is certainly interesting, but we would like to raise three points.

a) For quantizing large language models with 10B+ parameters, our understanding is that NF quantization (from QLoRA) and GPTQ --- methods we compared with --- are near state-of-the-art quantization methods. (While we show numbers from other methods, such as Omniquant, in the PTQ comparison table above, these are, as far as we know, preprints).

b) Applying fancier quantization techniques such as BRECQ/Adaround to large models is challenging. For example, for BRECQ, the OmniQuant paper [1] notes,

... BRECQ ... cannot be applied in models with billions of parameters because they are hard to optimize due to the huge solution space.

Similarly, the SpQR paper [2] stated that,

... BRECQ ... were designed for vision models or small-scale language models, with less than 100M parameters.

[1] https://arxiv.org/abs/2308.13137

[2] https://arxiv.org/abs/2306.03078

FlexRound is an interesting suggestion, but we were unable to find an open-source implementation of this. Hence, we focused our comparison on widely-used LLM quantization methods with open-source code, i.e., QLoRA and GPTQ-LoRA.

c) Finally, (and most importantly), we note that LQ-LoRA can work with generic quantization approaches! For example, during each step of the iterative algorithm, instead of using NF-quantization, we can use other quantization methods to minimize $\Vert \mathbf{X}\mathbf{W} - \mathbf{X}(\operatorname{quantize}(\mathbf{W}) + \mathbf{L}_1 \mathbf{L}_2) \Vert$. We focused on NF quantization because the quantization function is extremely quick (on the order of seconds), and it performs on par with GPTQ despite being data-agnostic. But we think it's worth highlighting this aspect of LQ-LoRA more, and we will discuss this further.

Hi there! Please let us know if the above response answered some of your questions in the original review, and please let us know if you have follow-up questions!

In particular, we want to highlight a few key takeaways from our response:

1. We included a baseline that uses OPTQ/GPTQ to quantize first before learning low-rank updates. We called this "GPTQ-LoRA" baseline "LREC" in the paper.
2. We performed additional experiments using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset. (Please see the above response for more details.)
3. The techniques in this paper are not specific to NF. We chose NF because of its superior quality. To be more concrete about this, we conducted additional experiments. (Please see the above response for more details, too.)

The authors referenced comments including (1) and (2) as below.

(1) ... BRECQ ... cannot be applied in models with billions of parameters because they are hard to optimize due to the huge solution space.

(2) ... BRECQ ... were designed for vision models or small-scale language models, with less than 100M parameters.

However, in the FlexRound paper, the experimental results of BRECQ are quite well for LLaMA 7B, 13B, and even 30B. In this sense, just referencing those comments seems not to be enough to reinforce the motivation of the paper. To confirm the validity of LQ-LoRA further, the paper needs to be more polished in my view.

Thanks to the reviewer for the comments! Please see our responses below.

Latency of The Iterative Algorithm

This is a one-time cost at the initialization and takes a fraction of the total training time. Each step of the iterative algorithm consists of SVD followed by quantization. Note that we use randomized SVD (rSVD) for speed consideration, and we run the algorithm for up to 100 steps (and we stop before this if the error starts to increase).

To be more concrete, we measured the runtime to process three of the LLaMA-2 7B's matrices with rank=64 and NF3 quantization. Please see the table below.

Comparison with GPTQ-based Methods

Thanks for the question! We actually did implement and compare against GPTQ-LoRA. We called this "LREC" in the paper following this paper (https://arxiv.org/pdf/2306.08162.pdf). We find that GPTQ-LoRA underperforms both QLoRA and LQ-LoRA (Table 1).

(We now realize that this naming caused some confusion. Thanks for pointing that out --- we will clarify this in the paper.)

Limitations from QLoRA

This is a very good point! But we want to point out that inference with quantized + low-rank is not necessarily slower, especially in batch = 1 setting. Smaller quantized matrices allow us to reduce data movement between SRAM and HBM of the GPU. For example, in the latest releases of bitsandbytes (the low-level engine behind QLoRA) [1], they claimed their matrix-vector multiplication between quantized matrix and dense vector (the one needed during inference) could be competitive and even outperformed dense matrix-vector multiplication.

[1] https://github.com/TimDettmers/bitsandbytes/releases

Integrating LoRA Layers Into Linear Layers

Given the potential inference speed-ups, we think that practitioners could use the "quantized + low-rank" models without merging. Such models could be just as fast (as discussed above) as merged dense models and more memory-efficient. In that regard, you could (loosely speaking) consider this as an alternative PTQ method.

Based on this potential of LQ-LoRA as an alternative to PTQ, we performed additional experiments by using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset. Specifically, we train the LQ-LoRA (Fisher, 2.75-bit, 64-rank) for one epoch on two C4 partitions and WikiText-2. We then further quantize the LoRA components themselves to 8-bit (NF8). Please see the Table below for the results, where the numbers other than LQ-LoRA are from OmniQuant [1]. Note that we use the term "effective bits" to denote bits per parameter that treat the LoRA components as "overheads" for comparing with methods without LoRA; LoRA components (with NF-8 quantization) amount to about 0.1-0.2 extra bits per parameter.

[1] https://arxiv.org/abs/2308.13137

These results indicate that LQ-LoRA can also be used as a state-of-the-art PTQ method. Given the promise of LQ-LoRA as a potential alternative to PTQ, we will include this comparison against PTQ results in the next iteration of the paper.

Novelty

Similarly, I would like to raise a query about the paper's novelty. While this method undeniably enhances the current approach (Q-LoRA), from a PEFT perspective, there could be superior methods, particularly concerning inference challenges. On the topic of novelty, I await the insights of fellow reviewers.

We want to reiterate three main contributions on top of QLoRA:

1. We adopt a matrix decomposition view of initializing the quantized and low-rank components of LoRA adaptation. (Note that QLoRA does not perform explicit matrix decomposition)
2. Our ILP formulation enables us to search for the mixed-configuration quantization that fits specific resource requirements (e.g., 2.75 bits per parameter).
3. We extended the two methods above to a data-aware setting by incorporating the sensitivities of parameters.

Hi there! Please let us know if the above response answered some of your questions in the original review, and please let us know if you have follow-up questions!

In particular, we want to highlight a few key takeaways from our response:

1. The iterative algorithm adds a small extra computation head --- a fraction of the total training time.
2. We included GPTQ-LoRA comparisons but referred to them as "LREC" in the paper.
3. Inference with quantized + low-rank is not necessarily slower, especially in batch = 1 setting (per the latest releases from QLoRA; kudos to them).
4. Practitioners could use the "quantized + low-rank" models without merging. Such models could be just as fast as merged dense models and more memory-efficient. Based on this potential, we performed additional experiments by using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset.

Thanks for the helpful review! Please take a look at our responses below.

Novelty with respect to QLoRA

We want to note that QLoRA does not perform explicit matrix decomposition. It performs quantization of the original weight matrix and learns additive low-rank updates. In contrast, we perform explicit matrix decomposition of the original matrix into low-rank and quantized components via a matrix reconstruction objective.

Ablations

How important is the ILP to LQ-LoRA? Can you show the performance of LQ-LoRA without the ILP?

One of the main usages of ILP is to flexibly quantize the model to meet specific memory constraints (e.g., 2.75 bits/parameter). That being said, LQ-LoRA is effective even without the ILP. For example, Table 4 shows LQ-LoRA without ILP. Here, we use NF3 quantization configuration --- NF3 reuses the configuration of the QLoRA's NF4 quantization with 3-bit first-level quantization.

Can you show the performance of the regular LoRA method (no quantization), and also quantization (at different bit-rates) without LoRA, in Table 2?

We want to note that the QLoRA paper already demonstrated little performance loss between regular LoRA and QLoRA (4bit). Hence, we can think of QLoRA (4bit) performance as a proxy for regular LoRA.

However, based on your suggestion/question, we performed additional experiments using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset. Specifically, we train the LQ-LoRA (Fisher, 2.75-bit, 64-rank) for one epoch on two C4 partitions and WikiText-2. We then further quantize the LoRA components themselves to 8-bit (NF8). Please see the Table below for the results, where the numbers other than LQ-LoRA are from OmniQuant [1]. Note that we use the term "effective bits" to denote bits per parameter that treat the LoRA components as "overheads" for comparing with methods without LoRA; LoRA components (with NF-8 quantization) amount to about 0.1-0.2 extra bits per parameter.

[1] https://arxiv.org/abs/2308.13137

These results indicate that LQ-LoRA can also be used as a state-of-the-art PTQ method. Given the promise of LQ-LoRA as a potential alternative to PTQ, we will include this comparison against PTQ results in the next iteration of the paper.

ILP for Rank Selection

This is a very good/interesting suggestion! We have looked into this by ILP-searching the bits and ranks together. One practical challenge is measuring the "storages/errors" of the ranks of low-rank components. This is important to trade off bits of the quantized matrices and ranks of LoRA. Notice that LoRA components will be fine-tuned; searching the ranks using errors at initialization will unnecessarily favor more bits at the cost of lower ranks. We might be able to formulate this as a bi-level optimization problem (first searching the ranks and then searching the bits), but we will leave this as future work.

Questions

Is the only difference between QLoRA+ILP, and LQ-LoRA, the initialization?

Yep!

Does the ILP budget, as well as the "bits per param" column, also consider the low-rank components?

This table does not take into account low-rank components since we use rank = 64 for our QLoRA and LREC (GPTQ + LoRA) baselines. However, the PTQ table above does take into account the contribution from the low-rank components to ensure fair comparison against other PTQ methods. We find that the LoRA components (if quantized) add about ~0.2 bits. We will clarify this further!

Suggestions

Thanks for the suggestions regarding the figures and tables! We will take these into account for the final version of the paper.

We thank the reviewer for their comments/questions. Please find our responses below.

Mixed Precision & Hardware

We note that our work is in the weight only quantization regime (as are QLoRA, GPTQ, etc.), where only the weights (and not the activations) are quantized to lower bits. The actual matmul is done in full precision after dequantization.

Concretely, we use just-in-time de-quantize the (quantized) matrix $\mathbf{Q}$, execute matrix operations (e.g., matmul(X, Q) ==> matmul(X, dequantize(Q))), and throw away the de-quantized matrix. This supports arbitrary quantization, and the main technical difficulty is how to do de-quantization quickly. We outlined more details in the appendix.

Although weight-only quantization cannot make use of (faster and more energy-efficient) lower-precision matmuls, weight-only quantization still has two practical benefits. First, at a macro level, having a smaller model (due to quantization) allows us to increase the batch size. This reduces the need for expensive data/model parallelism, which requires cross-device or even cross-node communication. Second, at a micro level, smaller quantized matrices allow us to reduce data movement between SRAM and HBM of the GPU. As an example, QLoRA's matrix multiplication implementation could be faster than dense matrix multiplication [1].

[1] https://github.com/TimDettmers/bitsandbytes/releases

Related paper on Joint Low-rank and Quantized Matrix Decomposition

Thanks for the suggestion; this is an interesting paper! We will discuss this paper in the related works section.

We primarily considered the cases in which $\mathbf{L}_1$ and $\mathbf{L}_2$ are floating-point parameters. This is necessary because we want to fine-tune these parameters. That being said, the techniques introduced in that paper paper might be helpful for other (future) use cases. For example, we could post-training merge $\mathbf{Q}, \mathbf{L}_1, \mathbf{L}_2$, and re-decompose them into three separate quantized matrices for inference.

Question 1

You are correct -- we will clarify this; thanks!

Question 2

You are correct in that the techniques in this paper are not specific to NF. QLoRA paper demonstrated the superior quality of such a scheme compared to classical integer quantization. Our preliminary experiments confirmed this, and hence, we chose NF.

We conducted the following experiment to be more concrete about NF vs INT. We ran LQ (i.e., Section 3.1) on LLaMA-2 7B's dense parameters with rank=64 and NF3/INT3/NF4/INT4 quantization configuration (i.e., b0=3/4, b1=8, b2=fp32, B0=64, B1=256), and measure the reconstruction errors (i.e., $||\mathbf{W} - (\mathbf{Q} + \mathbf{L}_1 \mathbf{L}_2) ||_2^2$). The table below shows that NF indeed performed better.

Question 3

This is the same $\delta$ in QLoRA, and we treated it as a constant for all settings. We will clarify further in the next iteration of the paper.

Question 4

We defined $\mathbb{Q}$ as the set of matrices that can be quantized (Equation 1 is a minimization problem, not an exact decomposition problem). Thanks for pointing out the confusion; we will clarify this.

Question 5

The middle figure represents the setting of many (Q)LoRA methods, in which the Low-Rank components are initialized to be zero. In that sense, $\operatorname{quantize}(\mathbf{W}) + \mathbf{L}_1 \mathbf{L}_2 = \operatorname{quantize}(\mathbf{W})$ for those methods.

Question 6

Good catch -- it's meant for "budget" but shared similar symbols with $B_0$ and $B_1$.

Question 7

We say a matrix $\mathbf{F}$ has homogenous rows or columns when either rows or columns of the matrix have identical values. We will clarify this.

Question 8

Table 4 used the quantization configuration of NF3/NF4. Specifically, they use the block sizes and second-level quantization as the original NF4, but with the first-level quantization set to 3/4-bit. Table 2, however, used ILP to search for the quantization configuration, hence the difference.

We made such decisions for Table 4 to remove the possible confounding variable of using ILP to search for the quantization configuration (i.e., we just wanted to see the effect of higher rank LoRA components). We will clarify!