LoRTA: Low Rank Tensor Adaptation of Large Language Models

We thank Reviewer aKMS for their thoughtful feedback and valuable questions. We are particularly grateful for recognizing the clarity of our presentation, the interesting connection between parameter-sharing methods and low rank tensor models, and the overall contributions of our work. Below, we address each of your concerns and provide updates to the manuscript based on your suggestions.

---

1. Performance Degradation on NLU Tasks

We acknowledge that on the submission there was a small performance gap observed on NLU tasks when compared to LoRA and VeRA which was a consequenco of using extremely low parameter counts together with parameters settings favourable to LoRA. We have now extended and revised our NLU experiments,

• Experiments with Larger Ranks: We conducted additional experiments by increasing the rank of LoRTA to $r=16$ and ( r=20).
• Applicability to Low-Resource Scenarios: We clarified in the revised manuscript that LoRTA is designed for scenarios where memory efficiency and low parameter count are critical, such as edge deployments or resource-constrained environments, or where massive amounts of concurrent adapters have to be managed.

In the newly added experimental settings for GLUE, which follow those from LoTR and LoreTTA, all methods are evaluated using 20-30 epochs, and the same hyperparameter grid-search is conducted for all methods.

In these settings, our method attains similar or slightly better performance while acheving a considerable reduction in parameter count.

Table1: Roberta Base scores and parameter count (including the linear classifier) in the experimental setting reported by LoReTTA

Table 2: Roberta Base scores and parameter count (not including the linear classifier) in the experimental setting reported by LoTR.

We thank Reviewer 69c8 for their thoughtful and detailed review. We are particularly grateful for recognizing the strengths of our work, including its clarity, simplicity, and relevance to a valuable problem in fine-tuning LLMs. Below, we address your feedback and provide updates based on your suggestions.

---

Addressing the Weaknesses and Suggestions

1. Mainstream LLM Evaluation Benchmarks

We acknowledge that mainstream LLM tasks like MMLU, MATH, and HumanEval are important for assessing LLM performance. However, we also note that these benchmarks are often omitted in the evaluation of competing PEFT methods due to their computational intensity.

• MMLU Exclusion: We excluded MMLU specifically due to compute constraints, as it involves a large number of tasks requiring significant resources to evaluate thoroughly.
• New Benchmarks Added: To address this gap, we have included new experiments for the LLaMa2-7B model on tasks such as SQuAD, DROP and Record in a low resource challenging setting proposed by by recent PEFT works. These results highlight LoRTA’s performance in diverse, task-specific settings and further validate its efficacy in the LLM domain.

2. Additional LLM Models

We recognize the value of evaluating on newer LLMs like LLaMa-3-70B and Mistral-7B. We prioritized the evaluation on Roberta and LLama2-7B due to their widespread use in the PEFT literature, to enable a more direct comparison with existing works. However, we will expand to include more comprehensive evaluations on larger LLMs as resources permit.

3. Performance Trends

We clarified in the paper that LoRTA is most beneficial in low-resource fine-tuning scenarios or when memory efficiency is critical, such as on edge devices or resource-constrained environments. The question of how do PEFT methods perform at high parameter counts is indeed a different yet interesting research direction (see e.g. LoRA vs Full Fine-tuning: An Illusion of Equivalence). You correctly observed that LoRTA's relative performance advantage diminishes as the parameter count increases in Figure 3. However, the same is not true for other experimental settings in the submission nor for other reported results in the literature, including LoRA, which report non-monotonic behaviour w.r.t. parameter count on a variety of settings.

---

We hope these updates address your concerns and demonstrate the broader applicability and benefits of LoRTA. Thank you again for your thoughtful feedback, which has significantly strengthened our work. We look forward to further discussions and insights.

Addressing Specific Questions

1. What is the trade-off between the number of parameters and performance?

   • Increasing the rank ins some cases improves performance and closes the gap with LoRA. However in LoRA and competing methods there is not a monotonic nor easy to predict relation between both due to the different optimization dynamics and inductive biases/regularization induced by constrained fine tuning methods. We clarified in the paper that LoRTA is most beneficial in low-resource fine-tuning scenarios or when memory efficiency is critical, such as on edge devices or resource-constrained environments. The question of how do PEFT methods perform at high parameter counts is indeed a different yet interesting research direction (see e.g. LoRA vs Full Fine-tuning: An Illusion of Equivalence).

2. Can LoRTA achieve better results with more parameters?

   • Yes, as shown in the additional experiments, increasing the rank of LoRTA improves performance and can match or surpass LoRA on most benchmarks.

3. How do training speed and memory usage compare to LoRA

   • Our main motivation to reduce adapter size comes mainly from improving task-switching and reducing storage requirements in practial applications where a large number --potentially thousands -- of adapters are used. CPU-GPU transfers required to load adapters can be costly in this scenario costly. For instance, compressing parameters further can allow thousands of customized models to coexist with the base LLM in GPU memory, significantly improving scalability and performance in multi-task environments.
   • As pointed out by the reviewer, the reduction in GPU memory requirements during training, which comes primarily from shrinked optimizer states, is typically negligible for parameter reductions beyond LoRA. Memory consumption in these settings comes primarily activations and caches stored during the forward and backpropagation. Further reductions in memory consumption could be attained by compressing activations or gradients exploiting the low rank structure of updates, or re-computing them on the fly. Our model has a smaller number of trainable parameters and the tensor CP structure could be efficiently implemented leading to both faster training and smaller memory usage. However, our current/naive implementation does not enjoy these benefits, and we leave efficient implementation of our model for future work. What is already attained due to the reduced parameter count is lower storage/disk space and thus less I/O time, which is relevant in applications with a massive number of adapters.
   • We have added this discussion along with a hardware profiling results to the manuscript, comparing our implementation of LoRTA against LoRA in huggingface PEFT. We highlight that the difference in resource consumption with our method is negligible, and that the gap in training time could be reduced by implementing our method more efficiently.

   rank		GPU Mem. (max)	FLOPS (avg)	MACs (avg)	Time
   4	LoRA	12.84	272	136	0.07
   	LoRTA	12.88	272	136	0.14
   64	LoRA	13.08	276	138	0.09
   	LoRTA	12.98	273	136	0.14

   Maximum GPU memory utilization, GB Flops and MACs and seconds per training step when finetuning Llama-7b on Copa dataset with batch size 1 on a single NVIDIA A6000 GPU.

4. How were hyperparameters tuned for LoRA in Preference Optimization and Protein Folding?

   • Hyperparameters for LoRA were tuned using the same procedure as LoRTA, described in Appendix E.

---

Additional Revisions Based on Suggestions

• Section Name: We agree that "X is all you need" is overused. We have renamed Section 3.
• Figures 2 & 4: We will move the training loss figures to the appendix, as suggested, and clarifiy that they are included for completeness rather than as evidence of superiority. However, we do find reporting train losses valuable, in order to analyze wether differences come from better optimization/fitting of the training objective or better greneralization (see e.g. LoRA vs Full Fine-tuning: An Illusion of Equivalence, for a deeper discussion).
• We have toned down performance claims, our method achieves a "parameter count reduction with comparable performance". The updated experimental section better supports this claim.

---

We hope these updates address your concerns and strengthen the clarity and impact of our work. Thank you again for your thoughtful feedback, which has significantly improved our paper. We look forward to further discussions and feedback.

We thank Reviewer g3FC for finding the idea interesting and clearly presented, but disagree on the lack of practical utility of our method. In that vein, we have updated the experimental section to further demonstrate that LoRTA con lead to parameter reductions with little to no performance trade-offs.

---

Evaluation and Claims About Performance

NLU Results and Parameter-Performance Trade-Off

We acknowledge your concern regarding the performance of LoRTA in the NLU setup. While LoRTA does exhibit a parameter efficiency advantage, we agree that the performance trade-off needed clearer quantification. We highlight that the small loss in performance reported in this task in the submission (at a two orders of magnitude parameter count reduction) is well within the variation due to hyperparameter settings, and using hyperparameters reported by LoRA and VeRa and not doing full hyperparameter searches represented a challenging setting for our own method. We have now included additional experimental settings, using higher ranks and following recent works DynaLoRA, LoreTTA and LoTR, which run the same hyperparameter gridsearch for all methods - including ours. In these settings, we have found our method performs comparably or slightly better than baselines, with at least 2x less trainable parameters.

Table1: Roberta Base scores and parameter count (including the linear classifier) in the experimental setting reported by LoReTTA[1]

Table 2: Roberta Base scores and parameter count (not including the linear classifier) in the experimental setting reported by LoTR[2], and Dynalora[4]

Other Experimental Settings

For the more computationally intensive settings of Instruction Tuning, Preference Optimization, and Protein Folding, we agree that comparing against more baselines would strengthen the argument, yet we were not able to do so due to compute constraints. However, comparing the proposed method with LoRA, which is one of the most widely used and established high-performance PEFT methods is a common practice in these settings (see e.g. LoReTTA). In adittion, we have now also added a SOTA method (LoReTTA) that outperforms LoRA for the Llama2-7B in the added experiments, as well as higher rank LoRA, which we show below:

Table 3: LLama2-7b performance on both SuperGLUE tasks and question answering tasks SQUAD and DROP. We follow hyperparameter settings in LoreTTA, where only 1000 train samples are used for each task to raise the difficulty.

We note that, to the best of our knowledge, LoTR has not been published at any peer reviewed venue and that it's tensor model is identical previous adapter models in vision (Fact-TK) which were discussed in the submission. Nonetheless, we have addressed your concern by expanding the discussion about LoTR and clarified the contributions of our work in the updated manuscript, which we summarize below.

---

Novelty Compared to LoTR

We acknowledge the similarity between our approach and LoTR, as both involve tensorizing attention weights for parameter-efficient fine-tuning. However, there are two key differences that distinguish LoRTA:

1. Tensorization Across Heads and Matrices:

   • LoTR represents updates as a third-order tensor $\mathbb{R}^{ML \times d \times d}$ and applies a Tucker2 decomposition to this tensor. This tensorization does not capture dependencies across attention heads or Q,K,V,P matrices.
   • In contrast, LoRTA uses a higher-order tensor $\mathbb{R}^{M \times L \times d \times d/h \times h}$, which models heads, layers, and attention matrices explicitly as separate modes. This design allows for parameter sharing across these dimensions, leading to greater parameter efficiency (better scaling) and better utilization of the tensor structure.

2. Use of CP Decomposition:

   • LoTR employs a Tucker2 parameterization, which introduces a quadratic dependence on the rank for the number of parameters.
   • LoRTA utilizes the CP decomposition, which scales linearly with the rank, resulting in fewer parameters for the same tensor rank. This efficiency is especially significant for large models and low-rank settings, as demonstrated in our experiments.

Parameter Scaling vs LoTR:

These differences are not only conceptual but also yield practical benefits, as shown in our updated experiments.

---

New Experiments Comparing LoRTA and LoTR

To substantiate our claims, we have included direct comparisons with LoTR in our revised manuscript, using the same settings as LoTR. These experiments demonstrate that LoRTA achieves competitive or superior performance with a smaller parameter count.

Results on GLUE Tasks

These results highlight that LoRTA outperforms LoTR across multiple tasks with a significantly smaller parameter count.

Responses to Specific Questions

1. How did you initialize the model?

   • We initialized the model using the pre-trained weights of the base model and initialized the CP decomposition factors in the same way as lora, using Kaiming-uniform initialization for all factors except for the last one which is zero initialized. We did not conduct additional ablations on initialization, which may lead to further performance improvements.

2. What is the test quality of the ESM-2 model without fine-tuning?

   • The baseline test quality of ESM-2 without fine-tuning is 0.655.

---

We hope this response clarifies the novelty and contributions of our work and demonstrates its practical advantages over related methods, especially LoTR. We look forward to further discussions and feedback.