LoQT: Low-Rank Adapters for Quantized Pretraining

We thank the reviewer for their thorough feedback. We appreciate the recognition of LoQT’s efficient handling of optimizer state memory, its effective quantization error management and its periodic update strategy.

1. Additional Finetuning Experiments

Excellent suggestion; we agree that more fine-tuning experiments should be included. We refer to the common comment where we describe fine-tuning of Llama 7B for a downstream task and continue pretraining for adaptation of the model to a new language. In both cases, LoQT excels.

2. LoQT is mainly based on GaLore with weight quantization

While our approach combines elements from existing methods like GaLore and others (see our detailed response to CPTS), we believe LoQT introduces significant new contributions, drawing on many related works besides GaLore in a non-trivial manner. We are the first to show that LoRA works well for pretraining given the right initialization, opening up further exploration in the LoRA literature. To our knowledge, we are also the first to present a method that can perform memory-efficient pretraining with most weights in 4-bit precision, resulting in a model that is already in 4-bit precision and thus does not require post-training quantization or quantization-aware training afterwards.

We show, theoretically, that GaLore can be rewritten to a LoRA format, where one of the low-rank factors is initialized as the gradient projection matrix P. We then prove that only updating B and keeping P and W frozen results in the same gradient updates as Galore. This enables a separation that enables quantization. However, we see that naive quantization of W and P leads to a significant decrease in quality (See Figure 1 in extra PDF). To tackle this, we developed a tailored error compensation approach, which is described in our response below. We also saw that during the later part of training, the errors we get from the quantization increasingly affect the loss. We show that less frequent updates over time help because we do fewer quantizations of W and P. This also speeds up training since it requires fewer SVDs during training.

Without the clear separations of frozen and trainable weights that enable us to use quantization, and without error compensation initialization of $B$ and without exponentially increasing update intervals, our method would not perform comparably to full training.

3. Quantization and Related Work

We chose NF4 simply because it has been shown to work well. Our goal is not to introduce a new way of quantization nor to thoroughly compare to the post-training quantization literature. However, we appreciate the suggestion of considering LoQT as a post-training quantization method and will consider this in more detail in the final manuscript.

Unlike other methods that adapt low-rank factors to quantization, such as LoftQ and LQLoRA, LoQT operates under different constraints. Specifically, we do not have the flexibility to freely choose both $A$ and $B$ factors because the matrix $A$ is already fixed as the projection matrix $P$ containing gradient information.

We will include a more thorough discussion and comparison of ApiQ, LoftQ, and LQLoRA in the final manuscript. See also our answer below for a direct comparison.

4. Difference between the LoQT error compensation mechanism and already published strategies such as LQLoRA and LoftQ

Both LoftQ and LQLoRA use the SVD of the quantization error to initialize the low-rank factors $A$ and $B$, aiming to minimize the difference between the quantized $W$ and $Q + AB$. The SVD gives the factorization $U \Sigma V^T$, where the top $r$ singular vectors in $U$ and $V$ are used to initialize the low-rank factors $A$ and $B$, respectively.

In contrast, LoQT takes a different approach due to the fixed nature of our low-rank adapter $P$ (analogous to $A$ in LQLoRA). Instead of applying SVD to the quantization error, we aim to minimize an objective where $W$, $Q$, and $P$ are fixed. We derive $B$ using the formula $B = P^+(W - Q)$, where $P^+$ is the Moore-Penrose pseudo-inverse of $P$ rather than the inverse. Incorporating information about the diagonal approximation of the Fisher information matrix (like in LQLoRA) into our objective could potentially reduce the error even further, a direction we are interested in exploring in future work. Additionally, LQLoRA proposes dynamically adjusting the bit-width.

5. Clarification on 7B LLaMA Experiments

We did not pre-train a 7B LLama model, as mentioned in the table caption of Table 4. We understand this is confusing, but we opted to include them for reference regarding the memory experiments presented in Figures 1 and 6. However, we have provided new results for continued pre-training on a 7B model, as described in the common response.

6. Finetuning without merging low-rank factors

Good suggestion. We have done this experiment showing how LoQT and LoQT-nq perform when not using the merging of factors while training. The results are shown in Table 1 in the extra PDF indicated with (No-Merge). We see that the results are slightly worse than those where merging is performed; however, the results are still better than LoftQ, LoRA, and Galore, showing that in the finetuning case, where we already have a pretrained model, we can omit to merge low-rank factors into W, and still get good results. This would allow W to be kept fixed and quantized while training only the adapters as suggested, at least for some tasks.

We thank the reviewer for their detailed feedback and for highlighting the strengths of our work.

1. More fine-tuning experiments

Good suggestion. We have included fine-tuning experiments of a 7B model in the common response. We adopt a Llama model for a new low-resource language and a commonly used benchmarking task, GSM8K, demonstrating that LoQT also performs well in downstream applications.

2. Comparison to prior methods

While our approach combines and builds on ideas explored in prior work like GaLore, ReLoRA, LoftQ, and NF4 quantization, we believe our work meaningfully extends these both in terms of how we deviate from the prior individual works and how we combine them.

We show that for the first time, LoRA can efficiently and successfully be used for pretraining while enabling the quantization of most weights with 4-bit precision, resulting in significant memory savings. The exponentially increasing update intervals, combined with the error compensation and gradient-based initialization, all come together holistically, where the combined benefits are larger than the sum of the parts.

1. Separation into Trainable and Frozen Weights

   • GaLore: Derives a low-rank projector using an SVD of the gradient. However, Galore does not separate trainable weights from frozen weights. In Galore, only the projector matrix $P$ is not trained, while the weight matrix $W$ is updated contiguously at every update step.
   • LoQT: We extend this by introducing a separation between frozen and trainable weights. Both $W$ and one of the factors (the projector $P$) are frozen while the new factor $B$ is trained. This further reduces memory for gradients (only low rank, whereas Galore uses "full rank" gradients) and enables pre-training within the LoRA framework. This then enables quantizing most of the model weights, as discussed below.

2. Gradient Based Initialization and Periodic Merging

   • ReLoRA: The $A$ and $B$ factors are initialized using Gaussian noise and zeros, and both are trained. ReLora then periodically merges the product of the factors into the weights of $W$ to explore a new subspace. However, ReLoRA requires full rank training for the first 1/3 of the time to be effective, limiting its usability on smaller GPUs.
   • LoQT: We build on ReLoRA by using clearly separated frozen and trainable weights. We show that to attain well-performing pretraining using just low-rank adapters, a gradient-based initialization of the LoRA factors is needed. This enables quantization approaches like NF4 from QLoRA while maintaining comparable precision to regular FP16 training. We only train a single low-rank factor per layer, which enables quantizing the weights of both $W$ and $P$, allowing for additional memory savings.

3. Quantized Pre-training

   • QLoRA: Works well for fine-tuning but not pre-training. However, implementing this with ReLoRA, without gradient-based initialization, would face similar problems, and quantization could exacerbate these issues.
   • LoQT: We use the specific NF4 quantization approach from QLoRA, as it has been shown to be superior to FP4 and INT4 quantization by a significant margin. We apply their approach to $W$ and $P$ by introducing error compensation and exponentially increasing update intervals:
     • Error Compensation Method: We introduce an error compensation method for initializing the $B$ matrix, incorporating the quantization error between $W$ and $Q$. This significantly reduces the performance gap between non-quantized and quantized models. We show (Figure 1 in the extra PDF) that simple quantization of $W$ and $Q$ creates a significant gap in model quality. The method reduces this gap by 72%. This is impossible with GaLore due to the continuous updates of $W$. While LoftQ also uses error compensation, their formulation operates under different constraints (see response 4 to Ekw9).
     • Adaptive Update Intervals: We vary the length of update intervals by exponentially increasing intervals for selecting a new projection matrix. This, in combination with the error compensations, reduces the error by 93%. This opens up further research opportunities into how the choice of update intervals affects learning and runtime.

In summary, our contributions nontrivially build on prior ideas of low-rank training and quantization, extending and combining them in ways that were previously not possible, with significant memory improvements and observed performance gains.

3. What is the reasoning for choosing NF4?

We chose NF4 since it performs better than FP4 (E2M1 and E3M0) and INT4 in related work. As shown in Table 2 (Page 7) of QLoRA, NF4 achieves a significantly better mean perplexity (PPL) than the other quantization methods. While our focus was primarily on reducing the memory footprint of pre-training, we acknowledge that INT4 could enable faster computations since it is directly supported on certain GPUs. However, to gain this benefit, one must do low-precision matrix multiplications, which we see as an interesting future direction.

4. Can you provide an analysis of memory savings with varying sequence lengths?

Good question. With larger contexts, the overall memory consumption is increasingly influenced by activations. Following prior work, our experimentation has focused on the setting of shorter context lengths (256 tokens). But as demonstrated in Figure 3 in the extra PDF, the benefit of LoQT does transfer to longer context lengths, enabling training of Llama 7B on consumer hardware with a context length of 2048 and 4096 on a 40GB A100 without activation checkpointing.

We thank the reviewer for the feedback and recognition of LoQT’s memory efficiency and its practical applicability. % for both pretraining and fine-tuning through gradient-based tensor factorization.

1. Size of models

We agree that training larger models than 1B would strengthen our work. In particular, the suggestion of fine-tuning on larger models. We have added experiments for fine-tuning a 7B parameter model to better showcase the potential of LoQT on larger scales. We show this successfully in the common response. See Tables A and B in the common response, and Figure 2 in the extra PDF.

While we would have liked to train models >1B from scratch, these require computational resources beyond those that have been available to us. As reported in Sec. 4.1 (Line 214), training a 1B model takes more than four days on four A100 GPUs for Galore, Regular FP16 training, and LoQT. Our aim was to validate our method across different model sizes to demonstrate that LoQT can achieve the same performance as regular FP16 training while maintaining most weights in a 4-bit quantized format and performing only low-rank updates. With this validation, we show that LoQT is effective in terms of quality while using much less memory.

2. Memory Efficiency in Figure 1

To clarify the differences between memory savings with layer-wise gradient computations and 8-bit Adam, we conduct an ablation experiment using a 130M parameter model. This experiment compares four settings: regular training, 8-bit Adam, layer-wise gradient updates, and a combination of 8-bit Adam with layer-wise updates, tested on Galore, LoQT, and regular FP16 training. Our results, illustrated in Figure 4 in the extra PDF, show that adding these memory-saving components introduces a small decrease in performance. Importantly, LoQT experiences a proportionally smaller decrease in performance compared to Galore and full training when combining both 8-bit and layer-wise updates. These results demonstrate that while memory savings come with some trade-offs, LoQT maintains good performance. In addition, due to the lower memory requirements of LoQT, we enable training of larger models without using layer-wise gradient computations and 8-bit Adam.

3. Clarification on LoRA for Pretraining (Tables 1 and 3)

We acknowledge the confusion regarding Table 3’s statement that LoRA cannot be used for pretraining. To clarify, while LoRA does not perform well for pretraining, it is not entirely unusable. As demonstrated by both Galore and ReLoRA, LoRA is comparatively less effective at pretraining. We will update Table 3 to indicate “Yes/Sort-of” and reference ReLoRA, which shows that even with a warm start (full training for 1/3 of the iterations), the difference in perplexity between LoRA and regular pre-training is still significant.

Generalization to other architectures and models

We agree that LoQT should work with any type of DNN using linear layers, such as vision transformers or state space models. To narrow the scope of our work and provide a more detailed analysis, however, we choose to focus on a well-studied auto-regressive language model and a bi-directional masked language model that have been commonly used as a basis in much of the related work. We believe that Llama 2 is representative of a large family of text-based Transformer LLMs, which overlap in most of the underlying architectural decisions. In Sec. 4.2 (line 231) and Tab. 2, we present results for a diverse set of model sizes ranging from 60M to 1B parameters within the Llama 2 architecture. In Sec. 5 (line 276) and Fig. 4, we provide an ablation study of each model component, as well as the effects of varying update interval sizes. In Sec. 4.3 (line 241) and Tab. 2, we show results for fine-tuning a bi-directional language model, the DeBERTaV3-base model on the GLUE tasks. All experiments are averaged over three seeds to ensure robustness.

Adding to this, we include more fine-tuning experiments of a pre-trained model in the common response. These include continued pre-training of the Llama2 7B model on a low-resource language and adaptation on a common benchmarking task (GSM8K) that confirms the memory saving and performance preservation using LoQT for adaptation of an existing model.

Having said that, we do plan to consider LoQT's generalizability in future work and thank the reviewer for pointing out this strength of LoQT. We will make sure to elaborate on this in the updated manuscript, including what needs to be kept in mind when using quantization for, e.g., vision models.