FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic

We appreciate the valuable comments from the reviewer. We also thank the reviewer for acknowledging our paper to be technically solid, the design is clear and thoughtful, and the paper is clearly written and the contribution is timely.

This rebuttal addresses the reviewer’s concerns and suggestions. Below is a brief summary of this rebuttal.

• W1, Q1-Motivation for FP8 is not fully convincing: We elaborate on the real-world scenarios (4 tasks from 6 papers) in the paper (line 111-118).
• W2, Q3.2-Why are QA-LoRA and IR-QLoRA slower than QLoRA despite having fewer trainable parameters, especially when QA-LoRA claims to outperform QLoRA in speed in its original paper?: The original claim is about inference speed. Our paper focuses on fine-tuning speed, which is determined by the computational overhead rather than the number of trainable parameters.
• Q2-Could the authors include experimental data showing the turning point where FP8 outperforms FP16 in practice for LoRA modules?: We present the experimental results and the turning point is around rank 8192.
• Q3.1-Is QLoRA implemented as in the original paper with 4-bit NF4 quantization?: Yes.
• Q4-Loss curve comparisons: We provide a training loss table of every hundred steps instead of a figure, due to the limitation of the conference policy.

We will improve our manuscript to address the reviewer. Please refer to the following for the detailed rebuttal.

W1, Q1: Can the authors elaborate on the real-world scenarios or hardware settings where such speed gains justify potential accuracy losses or increased complexity?

$\to$ For this, we can refer to the real-world scenarios provided in the paper (line 111-118), which trains up to millions of LoRA adapters, and therefore the speed up is especially beneficial. In lines 111-118, we present four scenarios that train a large amount of LoRA adapters for personalization [54, 50, 26], multitask learning [44], domain adaptation [11], and multilingual summarization. For example, PLoRA [54] assumes millions of users, and iLoRA [C1] conducts evaluation on a few hundred to 6 million users. Therefore, we believe our framework significantly provides benefits to these scenarios.

[C1] Kong, Xiaoyu, et al. "Customizing language models with instance-wise lora for sequential recommendation." NeurIPS (2024).

W2, Q3.2: QA-LoRA and IR-QLoRA, which have fewer trainable parameters than QLoRA, appear slower in Table 1, contradicting their original papers. Why are QA-LoRA and IR-QLoRA slower than QLoRA despite having fewer trainable parameters, especially when QA-LoRA claims to outperform QLoRA in speed in its original paper?

$\to$ We respectfully disagree that our results contradict the baselines’ original papers. The QA-LoRA claims that they are faster in the inference stage, while we report the fine-tuning speed.

In the original QA-LoRA paper, they claimed as “The major advantage of QA-LoRA, compared to QLoRA, lies in the inference stage where it is faster.” on page 5. This clearly shows that they mentioned the inference stage. However, in our target scenario, the fine-tuning speed is primarily determined by computational overhead, rather than the number of trainable parameters. As QA-LoRA and IR-QLoRA utilize auxiliary computations in addition to QLoRA, they are slower than QLoRA in fine-tuning, but they are faster in the inference stage due to optimization focused on inference efficiency. We believe this does not contradict the original paper, but aligns with their proposed algorithms.

Q2: Could the authors include experimental data showing the turning point where FP8 outperforms FP16 in practice for LoRA modules?

$\to$ The turning point of the LoRA adapter is rank 8192, which is an exceptionally large rank compared to common settings. We provide a detailed analysis across LoRA rank in Table R1.

Table R1: Efficiency analysis across various LoRA ranks

The experiments compare the computation time of LoRA adapters on FP16 and FP8 arithmetic. In addition, we provide the memory consumption of full fine-tuning (FT) and LoRA adapter to show the efficiency in relation to LoRA rank. Even if FP8 computation is beneficial in ranks 8192 and 16384, the memory consumption on those ranks exceeds that of full fine-tuning, making LoRA inefficient.

Q3.1: Is QLoRA implemented as in the original paper with 4-bit NF4 quantization?

$\to$ Yes, we used the NF4 quantization and also used the original code provided by the QLoRA’s authors.

Q4: Could the authors provide training loss curves for all quantized LoRA baselines compared to FALQON?

$\to$ Unfortunately, we could not present the figures due to the NeurIPS conference policy this year. Therefore, we provide a table showing the average training loss of every hundred steps (Table R2).

Table R2: Loss comparison across training steps

Overall, the results show the training curve of each method. The magnitude of the loss is similar to non-quantized FP16 LoRA, and the loss is reduced as the training continues. Also, please note that the training loss is not directly related to the performance of the downstream task (e.g., MMLU accuracy), due to the nature of the language models. Therefore, a larger training loss does not necessarily mean lower quality of the model.

We thank the reviewer for the helpful comments. We believe that our rebuttal well addresses the reviewer’s concern and potentially clarifies our contribution. We welcome further comments and would be happy to discuss them.

I really appreciate your thoughtful and thorough comments, which have resolved many of my concerns. However, one aspect remains unclear to me. Although QA-LoRA primarily focuses on inference speed, Table 2 in the QA-LoRA paper also reports that its fine-tuning time is considerably shorter than QLoRA. Could you clarify this point?

We genuinely appreciate the reviewer for the helpful comments and feedback. We are happy that the reviewer finds the proposed methods are valuable and sound.

We present this rebuttal to address the reviewer’s concerns and questions. Here is a brief summary of the contents.

• W1, W9-Would have been good to have a concise overview: We clarified the algorithmic overview and improved the explanation of key derivations, including diagram updates and explicit operation descriptions.
• W4-Request for detailed analysis for matrix dimension: We provide a detailed speed-up analysis across matrix sizes, showing that FP8 achieves empirical acceleration for dimensions above 4K.
• W11-Error bar for evaluations: We provide two-sigma error bars for the experiments, confirming consistency.
• Q1-What is special about FP8 compared to INT4/NF4?: We clarify that FP8 uniquely accelerates fine-tuning via fast GPU arithmetic, while INT4/NF4 primarily reduce inference memory.
• Typos and Writing suggestions: We have revised and clarified key sentences as suggested.

Here, we provide a detailed rebuttal.

W1, W9: Would have been good to have some sort of full picture overview or the key derivations together to get a concise overview. I think a more pedagogical approach could have been taken. Figure 2: For this to be easier to read it might be good to include the operations (addition and multiplication)

$\to$ Thank you for the suggestion! Please note that we have included the complete algorithm overview in Appendix Section B. Due to the page limits, we provided partial algorithms for each of the proposed methods in the main body of the paper.

Regarding Figure 2, we will add a modified version with the operations in the manuscript. Unfortunately, we could not post the revised figure in rebuttal, due to the system’s limitations. Therefore, we briefly mention how the change will happen. In Figure 2, activations ($\tilde X$ and $O$) connected to the weights ($\tilde W$, $\tilde A$, and $\tilde B$) with arrows are matrix multiplication. Weights and activations connected to (Quant) are quantization operations. Bold arrow in 3. Update figure is an addition for the weight update.

W2: Some sentences are very long or vague.

$\to$ Thank you for the constructive feedback. We will improve the sentences as follows:

• Line 92 $\Rightarrow$ Low-precision FP8 formats (e.g., E4M3 or E5M2) have a narrow representable range. Therefore, a higher-precision tensor $X$ (e.g., FP32 or FP16) must be scaled by a per-tensor scaling factor $s_X$. The scaled tensor $s_X \cdot X$ is then rounded to FP8, as follows:

• Line 149 $\Rightarrow$ LoRA fine-tuning commonly involves small-dimensional matrices. In such cases, the computational overhead of quantization outweighs the speedup.

W3 and W6: It would be good to make it clear you are talking about computational speed and not memory, as is often the focus of the low rank papers.

$\to$ We thank the reviewer for the great suggestion! We will further clarify this throughout the manuscript. For example, we will add “Unlike most of the prior LoRA-based methods that aim to reduce the memory cost, our method aims to speedup the fine-tuning of LoRA by leveraging the FP8 arithmetics, suggesting an algorithmic approach to maintain the accuracy while maximizing the speed.”

W4: Claims about cost of FP8<->FP16 in line 104-105. You say there will be a detailed analysis in section 4, but it is somewhat preliminary.

$\to$ We further provide a detailed analysis regarding the FP8 efficiency across LoRA rank in Table R1, comparing computation time and memory usage with FP16.

Table R1: Efficiency analysis across various LoRA ranks

The results indicate that FP8 incurs a slowdown for typical LoRA ranks under 4K, and only becomes faster than FP16 at extremely high ranks (over 8192), where memory consumption exceeds that of full fine-tuning. These findings clarify the limitations and trade-offs of FP8 for LoRA fine-tuning, and we will incorporate this analysis into the manuscript.

W5: In the background on LoRA: you may want to mention that the adapters can be downcast.

$\to$ Thank you for the suggestion. We will clarify that by adding “While prior methods focus on integrating LoRA to a quantized model, it is also possible to quantize the low rank adapters themselves, such as [C1].”

[C1] Guo, Hang, et al. "Intlora: Integral low-rank adaptation of quantized diffusion models." arXiv 2410.21759 (2024).

W7: I would say what your method does earlier, at least at a high level.

$\to$ We will add high-level description after the introduction, such as “We leverage FP8 arithmetics to accelerate the LoRA finetuning task, where we maximize the speedup by not keeping a physical LoRA weight. Instead, we use the quantization error space and shape it to imitate the effect of a LoRA update. Then, the update is migrated and merged to the original weight using top-k row selection, saving both the memory and the expensive update cost.”

W8: Typo: Line 84, you say $\tilde W$ is high precision

$\to$ We will correct that as follows: $DQ(\cdot)$ reconstructs the high-precision approximation of quantized weight ($\tilde W$).

W10: Worth mentioning LoftQ and LQLoRA on initialization from quantization error.

$\to$ Thank you for the suggestion. LoftQ is cited after Eq. 9 (line 195), and we will additionally reference LQLoRA in the revised manuscript.

W11: Error bars for evaluation.

$\to$ Here are the experimental results with two-sigma error bars.

Table R2: LLaMA-7B Experimental results with error bars

Following the NeurIPS paper checklist, we provide two-sigma error bars of our experimental results. Table R2 presents the MMLU accuracy on LLaMA-7B architecture. The experimental results show that our method shows consistent performance. Due to the character constraints, we provide partial results with error bars. However, we will put more experimental results with error bars in the manuscript.

W12: Excessive 244MB uncompressed code and not all .git folders were deleted.

$\to$ The immense size is from the large contents in the 3rd-party libraries (Transformers, PEFT, lm-evaluation-harness), which we modified for our framework. Please consider the large size of the code as our efforts to provide runnable code to the reviewers. Also, we found that the .git folder of the lm-evaluation-harness library is not deleted. However, it is from the public and popular GitHub repository. Therefore, it does not harm the double-blind policy of the conference.

Q1: What is special about FP8? Compared to say, INT4/NF4?

$\to$ FP8 quantization accelerates fine-tuning by enabling faster matrix multiplications, leveraging native GPU support for both weights and activations. In contrast, INT4 and NF4 quantization are primarily used for reducing inference memory and are typically weight-only. They do not speed up fine-tuning since computations still use high-precision arithmetic. Thus, FP8 uniquely enables significant training acceleration, which is the focus of our work.

Q2: Figure 1: What is behind the dashed lines, theoretical maximum, and pre-training speed up? What is the empirical setup?

$\to$ The theoretical maximum of FP8 is the Tensor Core throughput ratio of GPU (e.g., 330.3TFLOPS/165.2TFLOPS = 2.0 for RTX4090), pretraining speed up is reported from LLaMA-70B pretraining with 8xH100 GPUs, and the empirical speed up is measured speed of LoRA layers with TorchAO baseline on an RTX4090 GPU.

Q3: The sign of the error term flips from equation 8 to 9, keep it as negation?

$\to$ We respectfully argue that equations 8 and 9 are mathematically correct; therefore, the sign should be flipped. The sign is flipped because we did transposition of the error term $\Delta_Q W$ at equation 9: $DQ(\tilde W) + \Delta_Q W = W$ (Eq. 8) $\Rightarrow DQ(\tilde W) = W - \Delta_Q W$. However, if we misunderstood the reviewer’s question, please feel free to add a comment and elaborate on this.

Q4: Line 184-185: I don’t see another interpretation as possible, what do you mean?

$\to$ We mentioned that the key idea of melded LoRA interprets quantization error $\Delta_Q W$ as an additive tensor to the original high-precision weight $W$. We agree that it is a well-known interpretation of quantization error, and we apologize for the possible confusion.

We appreciate the reviewer for the constructive review. We believe our rebuttal addresses the reviewer’s concerns and better highlights the contributions of our paper. We would be happy to discuss if there are any additional concerns regarding our rebuttal.

We thank the reviewer for the constructive review. We are delighted that the reviewer acknowledged that the motivation and the central idea are interesting.

Here, we address the reviewer’s concerns and questions. This is a brief summary of the rebuttal.

• W1-Use cases for FP8 LoRA: We elaborate on real-world use cases (4 tasks from 6 papers) in the manuscript and further present cost comparison of FP16 and FP8 in rebuttal.
• W2-Accuracy comparison with FP16: We show comparison with FP16 LoRA and FALQON also provides comparable accuracy.
• W3.1/Q2-Detailed experimental settings: We provide more detailed experimental settings.
• W3.2-Why TorchAO causes OOM in 13B model?: TorchAO store weights as FP16 which takes more memory than FALQON.
• Q1-Computational overhead regarding top-k selection: We provide further analysis, and top-k selection incurs approximately 0.5% overhead of wall-clock time.
• Q3-Efficiency does not improve with increasing rank in the motivational study: The efficiency starts to improve after 4k rank (which is infeasible).

We will include this in the manuscript and thank the reviewer for the effort. The following is a detailed rebuttal.

W1: The motivation for FP8 format is not sufficiently justified. The paper would be stronger if it presented compelling use cases (e.g., scenarios where training thousands of adapters simultaneously makes the FP16 overhead non-trivial).

$\to$ We respectfully highlight that large-scale training of LoRA adapters is crucial to real-world applications such as personalization [54, 50, 26], multitask learning [44], domain adaptation [11], and multilingual summarization (see lines 111–118). For example, PLoRA [54] targets millions of users, and iLoRA [C1] evaluates up to 6 million users. In these settings, FP16 overhead becomes significant.

To further support this, Tables R1 and R2 provide a direct cost comparison on the MovieLens-1M [C2] scenario with 6,040 users, showing that FALQON reduces training time by months and cost by thousands of dollars compared to FP16 LoRA. This demonstrates the real-world benefits of our approach.

TABLE R1: Comparison of financial costs for fine-tuning LoRA

TABLE R2: Comparison of temporal costs for fine-tuning LoRA

[C1] Kong, Xiaoyu, et al. "Customizing language models with instance-wise lora for sequential recommendation." NeurIPS (2024).

[C2] Harper, F. Maxwell et al. "The movielens datasets: History and context." ACM TiiS (2015).

W2: The most significant weakness is the omission of the standard FP16 LoRA as a performance baseline in the results tables (Tables 1, 2, 3). The claim of maintaining "comparable accuracy" is only validated against other quantized methods.

$\to$ Thank you for this suggestion. We have included FP16 LoRA results below for direct comparison:

Table R3: Comparison with FP16 LoRA

The results show FALQON has competitive accuracy, even compared to FP16 LoRA. Please note that FP16 LoRA could not run on a single RTX4090 for LLaMA-13B due to its large memory consumption. Therefore, we run FP16 LoRA on an RTX A6000 GPU and only report accuracy without time per step (noted as OOM). Our focus on quantized methods is motivated by resource constraints common in practical deployments, but we agree that including FP16 LoRA enhances the comparison, and we will add these results to the manuscript.

W3.1: The experimental settings for baselines like QLoRA and TorchAO are ambiguous.

$\to$ We followed the original settings for each baseline: QLoRA and IR-QLoRA use NF4 for the backbone and FP16 for adapters; QA-LoRA uses INT4 for the backbone and FP16 for adapters. For FP8-based methods (TorchAO, Fishman et al., and ours), both the backbone and adapters use FP8. All experiments use a 32-bit paged AdamW optimizer. We will clarify these details in the revised manuscript. Thank you for the suggestion.

W3.2: The explanation for why TorchAO causes an OOM on the 13B model while FALQON does not is unclear. How come FALQON is more memory efficient than TorchAO?

$\to$ TorchAO stores master weights in FP16, so it does not reduce memory compared to FP16 LoRA. While TorchAO performs dynamic quantization for computation, FALQON maintains all weights in FP8, enabling significant memory savings. Consequently, FALQON can fine-tune the 13B model on a single RTX4090 GPU, which is not feasible with TorchAO or FP16 LoRA.

W4: The work's contribution is heavily weighted towards systems-level optimization and implementation efficiency rather than fundamental algorithmic advances in machine learning. While it involves algorithmic changes, the core novelty lies in overcoming a hardware-specific bottleneck. As such, the paper might be a more natural fit for a systems-focused conference like MLSys or DAC, rather than a general ML conference like NeurIPS.

$\to$ We respectfully disagree. The core idea of this paper is an algorithmic advance of the traditional quantization method by innovating the LoRA fine-tuning procedure. In addition, our target problem (the quantization bottleneck) is not a hardware-specific bottleneck, but an algorithm-induced slowdown. Because the quantization algorithm delivers the bottleneck, we improve the quantized training framework by modifying the LoRA algorithm. Lastly, quantization is one of the well-discussed topics in general-domain ML conferences such as NeurIPS. We believe our paper is beneficial to the general ML audience of the NeurIPS conference by providing practical values of efficient fine-tuning algorithms.

Q1: Could you provide an analysis of the computational overhead of this top-k selection and discuss how it impacts the overall wall-clock time speedup claimed by FALQON?

$\to$ The top-k selection step accounts for only about 0.5% of total training time (see Table R4), as it operates over $r$ rows (with $r \ll d$).

Table R4: Top-k selection overhead of LLaMA-7B and 13B models

Moreover, top-k selection reduces the update matrix dimension from $r$ to $k$, which lowers the weight update cost (See Figure 2 and Algorithm 3-line 6 in the paper.). As shown in Table R5, this leads to an overall speedup in step time:

Table R5: Comparison of time per steps with and without top-k selection

Thus, top-k selection has negligible overhead and actually contributes to overall training speedup.

Q2: The dimensions (e.g., rank r) for the micro-benchmark in Figure 1a are not specified.

$\to$ Thank you for the comments. The rank is 64, which is a default setting of all our experiments, unless mentioned otherwise.

Q3: The trend in Figure 1b, where efficiency does not improve with increasing rank, is counter-intuitive and contradicts the paper's own analysis without further explanation.

$\to$ The efficiency does improve when we increase the rank over 1024 (Table R6).

Table R6: Efficiency analysis across various LoRA ranks

The results show that the efficiency starts to increase after 1024 rank and becomes beneficial in ranks 8192 and 16384. However, the memory consumption on those ranks exceeds that of full fine-tuning, making LoRA inefficient at all.

We believe that this is not counter-intuitive as the quantization overheads scale in $O(n^2)$ while the computation of matrix multiplication scales in $O(n^3)$. Therefore, in the small-dimensional region (widely used rank 16-512), the efficiency does not seem to increase due to its exponentially scaling characteristics. We will modify the motivational study to the extended range and add this explanation.

We thank the reviewer for the valuable comments. We believe that this rebuttal effectively addresses the reviewer’s concerns and clarifies the paper's contributions. Also, we would be happy to address any additional comments the reviewer may provide.

We appreciate valuable feedback from the reviewer. We are encouraged that the reviewer finds our work novel and significant, with well-presented content.

We address the reviewer’s concerns and incorporate suggestions. Here is a brief summary of this rebuttal.

• W1-Suggesting evaluation of trade-off between performance and the number of LoRA adapters: We provide experimental results that present robust performance gains.
• W2-Extension to FP4 fine-tuning: We provide estimated speed up of FP4, where FALQON show 2.15$\times$ faster fine-tuning than FP16.
• Q1-Request for the rationale of freezing A matrix: Updating A matrix has small-to-no effects on the model quality [1], but incurs significant overhead.

We will incorporate these into the manuscript and thank the reviewer for the time. The following is a detailed rebuttal.

W1: The new approach seems to be valuable when multiple agents use multiple quantized LoRA adapters. It is suggested that multiple LoRA adapter fine-tunings are evaluated on a specific benchmark in the paper, demonstrating the trade-off between performance and the number of quantized LoRA adapters.

$\to$ Thank you for the comments. We believe the reviewer is asking for scalability on training multiple LoRA adapters at the same time. To address this, we provide an analysis of the trade-off between performance and the number of LoRA adapters in Table R1.

Table R1: Trade-off between performance and the number of LoRA adapters.

This experiment directly evaluates the scenario mentioned by the reviewer, demonstrating that FALQON consistently outperforms the baseline in terms of time per step, even as the number of LoRA adapters increases.

However, when we say “training numerous adapters” in our paper, we refer to running multiple independent fine-tuning jobs, each producing a single adapter. As far as we know, there is no use case to train multiple LoRA weights attached to the same backbone, simultaneously. Thus, we also provide experiments on training multiple LoRA weights independently. Table R2 reports the total training time required for such large-scale personalization, using the MovieLens-1M [C2] dataset scenario from [C1], which involves 6,040 users.

TABLE R2: Comparison of temporal costs for fine-tuning LoRA

These results confirm that FALQON provides significant temporal savings for both concurrent and large-scale fine-tuning scenarios.

We hope these results address the reviewer’s concerns. If further clarification or discussion is needed, we welcome additional questions in the discussion phase.

[C1] Kong, Xiaoyu, et al. "Customizing language models with instance-wise lora for sequential recommendation." NeurIPS (2024).

[C2] Harper, F. Maxwell, and Joseph A. Konstan. "The movielens datasets: History and context." ACM TiiS (2015).

W2: It would be helpful to add FP4 fine-tuning baselines.

$\to$ Thank you for this helpful suggestion. To address it, we provide a comparison in Table R3 below. The FP4 speedups are estimated by applying the FP4/FP8 matrix multiplication throughput ratio, as specified in NVIDIA GPU documentation and available hardware benchmarks:

Table R3: Speed up Comparison for FP8 and FP4 compared to FP16

These results suggest FP4 could provide further speedup over FP8 for both TorchAO and FALQON, assuming similar implementation efficiency. However, current hardware and software support for full FP4 fine-tuning is still limited. We are actively seeking access to FP4-capable resources and will report empirical results as they become available.

Additionally, we would like to clarify that prior works [1,2,3] primarily focus on weight-only FP4 quantization for inference rather than end-to-end fine-tuning. Most of them do not speed up fine-tuning since computations still use high-precision arithmetic or do not support backward or update of weights. Recent papers such as [C3] and [C4] do address FP4 fine-tuning, but they target full-rank LLM training and focus mainly on large-dimensional matrix multiplications. In these scenarios, quantization overheads remain significant, and we believe FALQON’s approach would continue to offer benefits even in the FP4 regime.

[C3] Castro, Roberto L., et al. “Quartet: Native FP4 Training Can Be Optimal for Large Language Models.” arXiv preprint arXiv:2505.14669 (2025).

[C4] Chmiel, Brian, et al. “FP4 All the Way: Fully Quantized Training of LLMs.” arXiv preprint arXiv:2505.19115 (2025).

Q1: Could you explain why gradient computation for matrix A is not required?

$\to$ It is because updating matrix A does not significantly contribute to model quality [49], while slowing down the training. The earlier work, LoRA-FA [49], empirically showed that freezing matrix A has little effect on the final quality of fine-tuning.

In addition to LoRA-FA, we conduct further experiments showing the effect of freezing matrix A in Table R4. The results empirically verify that we may freeze matrix A during fine-tuning without sacrificing much of the model quality. Also, computing the gradient of matrix A affects the training speed, showing approximately a 16% slowdown (13B). This is because gradient of matrix $A$ ($\frac{\partial \mathcal{L}}{\partial A}$) is computed as $B^{\top}\frac{\partial \mathcal{L}}{\partial O} x^{\top}$, which cannot be pre-calculated unlike matrix $B$. We thank the reviewer for their insightful comments, and we will clarify this in the revised version of the manuscript.

Table R4: Comparison of training speed and model quality regarding matrix A update

[49] Zhang, Longteng, et al. "LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning." arXiv preprint arXiv:2308.03303 (2023).

We thank the reviewer for their insightful comments. We believe this rebuttal addresses the reviewer’s concerns and further clarifies our contribution. We welcome further questions or comments and would be happy to discuss them.