LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

Q1: “The performance of LoRA-XS is highly dependent on which singular vectors (top, middle, or bottom) are retained for each module. For example, the self-attention layers may perform well even when only a small fraction of the top singular values is kept, but the output dense layers might require a larger portion of the singular spectrum to maintain good performance. This sensitivity suggests that careful tuning is necessary for optimal results.”

• Thank you for raising the point regarding the sensitivity of LoRA-XS to the singular vectors retained for different layers. We agree that the performance of LoRA-XS is influenced by the specific choice of singular vectors, and we will highlight this potential for improvement in the revised version of the paper.

Q2: “LoRA-XS relies on using SVD to construct the low-rank matrices and then inserting a trainable matrix. How to initialize this small matrix and how to choose the rank?”

• To clarify, we initialize the r×r matrix using a Gaussian distribution N(0,σ^2), where σ is set to a small value (specifically σ=10^(−5)) for all our LoRA-XS experiments. Similar to LoRA, this initialization ensures that we essentially start the fine-tuning from the pre-trained model weights.

• Regarding the choice of r, LoRA-XS is designed to be flexible, allowing users to select the rank based on available memory and computational resources. We agree that dynamic rank selection is an intriguing idea that could be explored to optimize performance further, depending on the task or model size. This is an area for future research, and we will include this potential direction in the updated paper.

Q3: “What is the time overhead of the initial computation of the SVD for the weight matrices of the base model, particularly for very large models?”

• Thank you for highlighting the computational cost of SVD. To address this, we note that SVD is a one-time computation for each model weight matrix, which is considerably faster than full fine-tuning and is offset by LoRA-XS's efficiency gains during adaptation.

• To quantify this, we conducted experiments on an H100 GPU using RoBERTa-large across 24 layers. Specifically, we compared the total fine-tuning time with the SVD initialization time for all matrices on the SST-2 (20 epochs) and MRPC (50 epochs) tasks with ranks of 4 and 25.

The results are as follows:

Notation:

Total Fine-Tuning Time (in seconds) – TF_T

SVD Initialization for all adapters (in seconds) – SVD_T

• These results demonstrate that SVD costs are minimal relative to the overall fine-tuning time, confirming that the initialization step does not significantly impact computational efficiency. We will include these findings in our paper for further clarity.

W1 & Q1: “Computational cost for SVD for each pre-trained weight matrix.”

• Thank you for highlighting the computational cost of SVD. To address this, we note that SVD is a one-time computation for each model weight matrix, which is considerably faster than full fine-tuning and is offset by LoRA-XS's efficiency gains during adaptation.

• To quantify this, we conducted experiments on an H100 GPU using RoBERTa-large across 24 layers. Specifically, we compared the total fine-tuning time with the SVD initialization time for all matrices on the SST-2 (20 epochs) and MRPC (50 epochs) tasks with ranks of 4 and 25.

The results are as follows:

Notation:

Total Fine-Tuning Time (in seconds) – TF_T

SVD Initialization for all adapters (in seconds) – SVD_T

• These results demonstrate that SVD costs are minimal relative to the overall fine-tuning time, confirming that the initialization step does not significantly impact computational efficiency. We will include these findings in our paper for further clarity.

W2 & Q2: “Ambiguous theoretical role of the r×r matrix.”

• Thank you for prompting us to elaborate on the theoretical role of the r×r matrix in LoRA-XS. The r×r matrix (initialized with Gaussian noise) serves as a task-adaptive layer. By fine-tuning this matrix, LoRA-XS effectively performs latent editing within a constrained r-dimensional subspace, driven by task-relevant components. We will expand on this in the paper to provide more insight into the functional role of this matrix.

W3: “A minor concern : The approach lacks novelty.”

• We appreciate your perspective on LoRA-XS’s simplicity, which we see as a strength. This approach offers both ease of implementation and practical utility. Looking forward, we anticipate growing demand for personalized models, which would enhance LoRA-XS’s usability even further. By storing only the small r×r matrices and computing the fixed SVD later as needed, LoRA-XS allows efficient storage and scalability.

W1: Limited Applicability of Memory Savings

• Thank you for pointing out the practical scope of LoRA-XS's memory savings. To clarify, the primary advantage of LoRA-XS is not limited to accommodating millions of adapters within a single model instance but rather to efficiently storing millions of personalized model variants. We envision a future of highly personalized AI, where LoRA-XS could store minimal r×r matrices for each personalized model, ideal for applications in education, healthcare, or other settings where per-user or per-task models may be beneficial. We will clarify this use case in the final version.

W2: Computation Overhead

• We appreciate your question regarding computational overhead. To address this, LoRA-XS does not introduce additional inference overhead as it merges directly into model weights, similar to LoRA.

• Regarding training time: to demonstrate the minimal effect the extra multiplication has on LoRA-XS training time, we trained RoBERTa-large using both methods on the MRPC task for 10 epochs, for three different ranks on an H100 GPU. As you can see in the following table, the runtime is minimally affected as the extra multiplication is performed over the latent dimension.

W3: Inconsistent Performance Across Tasks

• Thank you for discussing LoRA-XS performance across tasks. LoRA-XS is designed for resource efficiency, making it particularly useful in settings where storage and memory limitations are a priority. Due to computational constraints, we performed a limited hyperparameter search for mathematical reasoning experiments on the 7B models, with most parameters set according to prior works. We believe additional tuning would close the performance gap with LoRA, as seen in tasks like GLUE, GSM8K, and commonsense reasoning, where LoRA-XS consistently performs competitively while being more parameter-efficient. We will further elaborate on this point in the revised paper.

W4: Lack of Guidance for Hyperparameter Selection

• We recognize the importance of guidance on selecting an optimal r value. We see this as a key topic for future research, as optimal rank selection likely depends on both task complexity and model size. LoRA-XS’s strength lies in its flexibility, allowing r to be selected to meet specific memory constraints. We see the rank selection as a promising direction for future research, and we will address this in the paper.

Q1: “Gemma 7B base model's performance on MATH is 24.3 according to Hugging Face (source). Why is the Full FT performance of 22.74 even lower than the base model?”

• Thank you for this question. The number we reported was taken from previous work, and we conducted our experiments using a 0-shot prompting setting. The scores reported on Hugging Face, however, are based on a 4-shot setting. We will clarify this in the paper.

Q2: “In Table 3, LoRA-XS underperforms compared to LoRA in the MATH task. Have you tried using larger ranks for LoRA-XS to improve performance?”

• Due to computational limitations, we tested LoRA-XS with a single hyperparameter configuration. However, we believe that a more extensive hyperparameter search would help narrow the performance gap.

Q3: “In Appendix C.2, you train for two epochs on MetaMathQA for both LoRA and LoRA-XS, while the typical fine-tuning epoch number is usually 3. How did you choose this hyperparameter, and is it optimal for LoRA or LoRA-XS?”

• Thank you for raising this point. In order to have a fair comparison with the baseline, we followed the same number of epochs as those used for the baseline training. We will clarify this in the revised paper.

Q4: “In Appendix D, you find that the top subsets of singular vectors in RoBERTa-large retain the most task-relevant knowledge on MRPC, SST2, and MNLI. Does this result hold for different models (e.g., larger models like LLaMA3-70B) and other task scenarios (such as coding and math)? Could you provide some theoretical insights or analysis?”

• Due to computational constraints, we focused our ablation study on RoBERTa-large. However, the strong performance results of LoRA-XS across diverse, larger-scale benchmarks in the main paper indicate that our method is indeed generalizable to even larger models.

W1:

• Could you please clarify what you meant by "learn the tra"? We want to address your point accurately.

W2 & W3: “The results of full parameter fine-tuning for Mistral (7B) and Gemma (7B) are directly taken from other resources, and the performances are lower than LoRA FT, which is a little bit strange.” “LoRA fine-tuning results for LLaMA series are also directly taken from other resources.”

• For this experiment, we indeed used the same script as the prior work to train our LoRA-XS models to ensure a fair comparison with LoRA and full fine-tuning baselines. We acknowledge the point about the possible variations in platform conditions when using results from prior resources. However, due to resource constraints, similarly to other works, we’ve used the results from previous reliable studies. We will ensure that all sources are explicitly cited in the final version to clarify this.

W4: “LLM with LoRA-XS requires more space to store U, V, and Σr, and requires one more matrix multiplication compared to LoRA.”

• Thank you for raising this question. In LoRA-XS, only the small trainable r×r matrices are fine-tuned, while U and V remain fixed during fine-tuning. Furthermore, as LoRA-XS merges these trainable weights directly into the model during inference, there’s no need to retain additional matrices after fine-tuning (no added computational overhead during inference).

• If storing the adapters would be needed, we can store only the r×r matrices, as U and V, derived from SVD, remain constant and don’t require separate storage for each task or personalized model.

Q1: “I know the full parameter fine-tuning may rely on too much computational resource, could the authors do their best efforts to do LoRA fine-tuning on LlaMA 2-7B and LlaMA3-8B evaluation?”

• Due to computing limitations, we aimed to ensure a fair comparison by directly applying LoRA-XS modules for LlaMA 2-7B and LlaMA3-8B into the existing code-base from previous work (DoRA paper). We will make sure to mention this in the paper.

Q2: “Directly doing SVD on weight matrices usually shows unignorable errors; could the author explain why the proposed method (i.e., applying a weight W) could reduce this error?”

• Thank you for this question. While it’s true that direct SVD on weight matrices can introduce approximation errors, LoRA-XS is designed to mitigate this. By applying a small trainable matrix R between the fixed singular matrices U and V, LoRA-XS provides an adaptation that compensates for any discrepancies introduced by the SVD approximation.

Q3: “Could the author do the evaluation for efficiency in terms of both HBM costs and the computational consumption by applying one more Matmul?”

• We appreciate your question. LoRA-XS does require an additional matrix multiplication for R, but as this multiplication is done over the latent space, it’s computationally cheap and thus minimally affecting the model’s forward pass time. Moreover, the fixed U and V matrices are non-trainable, so they do not increase memory usage for gradients or optimizer states. At inference, we can merge R directly into the model weights, eliminating any extra computational or memory overhead. We will clarify this in the paper.

• To demonstrate the minimal effect the extra multiplication has on LoRA-XS training time, we trained RoBERTa-large using both methods on the MRPC task for 10 epochs, for three different ranks on an H100 GPU. As you can see in the following table, the runtime is minimally affected as the extra multiplication is performed over the latent dimension.

W1: "The discussion on related works such as including [1-2] can be further improved, given the rapid progress on LoRA-based parameter efficient fine-tuning of LLMs."

• We agree that recent studies in parameter-efficient fine-tuning are relevant to this work. Thank you for suggesting additional references. We will incorporate [1] and [2] in our related work section and also expand our discussion on LoRA-based fine-tuning methods.

W2: “The representation ability of LoRA-XS seems to be weaker than LoRA since the space resulting from LoRA-XS is much smaller than LoRA regarding the dimension. Could this make LoRA-XS easier to overfit than LoRA? Besides, is it possible for LoRA-XS to be harder to learn on more difficult tasks than LoRA?”

• Thank you for discussing this point. We believe that the reduced parameter count in LoRA-XS could actually mitigate overfitting, as the model has fewer parameters to memorize the downstream tasks’ training examples. Our experiments also do not show evidence of an overfitting issue, as demonstrated across various models and tasks. We will add this clarification to the paper.

W3: “Could the authors showcase the results on LLMs larger than 7B to further demonstrate the effectiveness of the LoRA-XS.”

• We appreciate your suggestion to test on larger models. Due to resource constraints, following other similar works, we focused on multiple 7B and 8B models to demonstrate the generalization capabilities of our method.

W4: “The proposed Theorem 3.1 is rather general and only related to the derivation of LoRA-XS. Therefore, it feels a little overclaimed by writing it as a Theorem since the theoretical contributions are limited.”

• Thank you for your feedback on Theorem 3.1. We believe it provides insights that apply not only to LoRA-XS but also to other low-rank (parameter-efficient) adaptations. We will revise the phrasing to emphasize the theorem's relevance more accurately. Additionally, Theorem 3.1 offers valuable insights into the potential applicability of LoRA-XS (see Table 4 and Appendix F for more details). It helps explain why parameter-efficient adaptation of large language models (LLMs) can perform effectively for various fine-tuning tasks. If LoRA-XS shows lower performance on a specific task, it may indicate that this task diverges from the pre-training objective, suggesting that increasing the rank could improve adaptation. We will clarify this in the paper.

W1: Since the input is mapped to a subspace of W and adjusted in scale within that space, if the distribution of the dataset used for fine-tuning differs significantly from that of the pre-training dataset, it may not adapt adequately.

• We evaluated LoRA-XS on diverse datasets, including GLUE, GSM8k, MATH, and eight commonsense reasoning tasks, and observed competitive or superior performance compared to LoRA, even on datasets that diverge from the pre-training distribution. For example, on SST-2 (a sentiment classification task), LoRA-XS initialized with random SVD performed only slightly better than when initialized with the SVD of pre-trained weights, indicating that it adapts well even when tasks differ somewhat from the pre-training objective (Table 4 and Appendix F).

• In cases where LoRA-XS might underperform we can increase the rank to allow for a subspace with higher dimensionality. These experiments lie outside the scope of our current work but we agree it would be an interesting direction to explore dynamic rank allocation in the future.

W2: In Table 3, although the computational efficiency is promising since the learnable parameters are drastically reduced, the performance decrease is noticeable. For instance, in the case of MATH dataset in Gemma model, the performance decreases from 31.28 when LoRA is used to 27.62 when LoRA-XS is applied. It is questionable whether this performance can be said to be comparable to the parameter efficiency advantage.

• For our mathematical reasoning experiments, due to computational constraints we performed limited hyperparameter search for the 7B models and set most parameters according to prior works. We believe additional tuning would narrow the gap with LoRA, as seen in other tasks like GLUE, GSM8K, and commonsense reasoning, where LoRA-XS performs competitively while being more parameter-efficient.

W3: In line 280, the authors refer to “(to eigenvalues).” However, in general, the pre-trained weights may not always be applicable for eigendecomposition, meaning that eigenvalues may not always exist.

• Thank you for catching the terminology inconsistency (L280). We intended to refer to the first r singular values, not eigenvalues. We will correct this in the manuscript. Regarding implementation, we use randomized SVD to decompose the pre-trained weights.

Q1: The results reported in Table 1 exclude QQP and MNLI datasets from the GLUE task. As far as I know, these two datasets are significantly larger than other reported datasets. Is there any reason why you did not conduct experiments on these two datasets? Personally, I think it is because larger datasets are complex and require more ranks and parameters. Can you conduct additional experiments on these datasets?

• Due to resource limitations, we prioritized a representative subset of GLUE and did not include QQP and MNLI, which are larger and would require additional resources for extensive hyperparameter tuning. Recognizing the importance of these datasets, we will aim to add results for QQP and MNLI in the final paper.

Q2: “Although the learnable parameters are greatly reduced, the matrix multiplication computation still seems similar to LoRA. I am wondering about the results for the actual runtime/GPU peak usage of the model.”

• Regarding runtime: During inference, as we merge LoRA-XS parameters into the pre-trained weights, the inference will be similar to the base model (similar to LoRA). During training time, compared to LoRA, we do have an extra matrix multiplication, which is performed on the latent space, which makes it computationally cheap. To demonstrate the minimal effect the extra multiplication has on LoRA-XS training time, we trained RoBERTa-large using both methods on the MRPC task for 10 epochs, for three different ranks on an H100 GPU. As you can see in the following table, the runtime is minimally affected as the extra multiplication is performed over the latent dimension.

• Regarding fine-tuning memory usage: We performed all our experiments without quantizing the pre-trained model weights. This causes the memory footprint improvement for both LoRA and LoRA-XS negligible, although our method uses less memory. We expect to have a considerable improvement in memory footprint when training LoRA-XS with quantized weights, but we leave this to future work.

We would like to point out that LoRA-XS’s application mainly lies in the scenario where we need to deploy multiple personalized models, and we can store them using significantly less memory (store only r x r matrices as SVD can be computed from pre-trained weights).