We thank the reviewer for their detailed review.

• Q1: VeRA already pointed out the performance saturation as the rank of LoRA increases.

Thank you for bringing this to our attention, we have added a citation to VeRA line 46.

To the best of our knowledge, while VeRA noted that LoRA's parameter efficiency decreases with increasing ranks (Section 3.2, Paragraph 1 in the VeRA paper), it does not identify low rank as an inherent limitation. VeRA primarily focuses on minimizing trainable parameter count beyond rank-1 LoRAs (Section 2, Paragraph 2 and Contributions, Page 2, Point 1 in the VeRA paper) and on reducing disk storage requirements for fine-tuned weights (Section 3.2, Paragraph 2 in the VeRA paper).

Our work identifies low-rank approximation as a limitation and proposes to decouple it from our parameter-efficient fine-tuning formulation. We propose a full-rank formulation, while maintaining the same parameter and memory efficiency. This allows our method to combine parameter-efficiency and full-rank to achieve strong performance.

• Q2: RandLoRA is longer to train than LoRA, what are possibilities for improvement ?

We acknowledged the reviewer's observation (L547-555) regarding RandLoRA's primary limitation: increased training time proportional to model size. To mitigate this, future enhancements would focus on implementing matmul-free matrix combinations as an efficient use of the ternary sparse random bases introduced in Section 6.4.

Notably, our experiments (Table 4) demonstrate that sparse random matrices containing {-1, 0, 1} values maintain performance. An efficient implementation would thus simplify the matrix product of $B$ by $\Lambda A\Gamma$ to simple aggregations, eliminating floating-point arithmetic [1]. Although CUDA kernels for such operations are currently unavailable [2], their future development would significantly accelerate RandLoRA training. This optimization would alleviate RandLoRA's dominant limitations and we hope it inspires further community contributions.

We revised Section 6.6's first paragraph to incorporate this discussion on matmul-free approaches. Additional strategies for overcoming RandLoRA's limitations are discussed in Section 6.6, including faster convergence via optimal random bases.

• Q3: RandLoRA becomes more accurate for larger amounts trainable parameters but LoRA is still competitive for smaller parameter counts.

We concur with the reviewer's assessment. To address their suggestion, we have explicitly clarified Line 74 that while LoRA can be preferable for scenarios where very high parameters-efficiency is required, RandLoRA presents an alternative in situations demanding larger parameter capacities. This typically includes challenging tasks or scenarios where available zero-shot models require significant adjustments.

[1] Li, Ping, Trevor J. Hastie, and Kenneth W. Church. "Very sparse random projections."ACM SIGKDD international conference on Knowledge discovery and data mining. 2006.

[2] Zhu, Rui-Jie, et al. "Scalable MatMul-free Language Modeling." arXiv preprint arXiv:2406.02528 (2024).

• Q4: Comparisons with further related works that propose improvements over LoRA.

We agree that an exhaustive comparison is desirable to improve understanding of the state of the field and promote fairer research.

Additions to Section 2.1

We have added the proposed recent works to Section 2.1 and point out that they showcase how important the parameter-efficient field is to the community. We kindly point out that to the best of our knowledge, only DoRA, FLoRA and SVFT have appeared in peer-reviewed conferences at this time.

Relation between RandLoRA and the suggested works

We note that RandLoRA's uniqueness, expressed through its full-rank parameter-efficient updates make it largely orthogonal to the suggested works. RandLoRA should then be considered more as an add-on on top of existing methods build around LoRA than a strict alternative.

For instance, DoRA's update is represented as $m \times \frac{LoRA}{\| LoRA \|}$ (i.e. normalizing then scaling the updates), implying RandDoRA would naturally extend to $m \times \frac{RandLoRA}{\| RandLoRA \|}$. The multiple upscaling matrices of Hydra-LoRA could be computed using RandLoRA's strategy. LoRA-XS or SVFT could be used as alternatives to estimate SVD slices in Equation 6.

Generally, we propose that novel methods that enhance LoRA's convergence without altering its rank can reasonably be expected to yield similar improvements in Rand-X variants.

Further comments on the related works

We identify that Pissa and DoRA are particularly relevant to the comparison as similarly to RandLoRA they propose improved results over LoRA for the same parameter count.

Although relevant to the understanding of the research landscape, the other algorithms are less suited to the comparison as they serve goals miss-aligned to RandLoRA's: SVFT and LoRA-XS aim to greatly reduce the amount of parameters in LoRA with minimum decreases in accuracy, FLoRA is not parameter-efficient as it focuses on gradient rather than weight compression and Hydra-LoRA learns non-mergeable weights selected through a Mixture-of-Expert strategy, thus creating a new model at inference time.

Experimental results

For completeness of the experiments and following the reviewer's suggestion, we are now working on reporting DoRA results for the commonsense reasoning tasks on LLama3-8b in the newly-added Table 2.

We will continue to update Table 2 with DoRA results as they come in to facilitate direct comparison and provide a clearer understanding of RandLoRA's position within the research landscape. We are also aiming to provide RandDoRA results by the camera-ready to support our point about compatibility.

We thank the reviewer again for their in-depth review. Please let us know if you have further concerns you would like addressed.

We thank the reviewer for their review of our work.

• Q1: How can we guarantee that the random matrices follows a Gaussian or uniform distribution ?

Thank you for raising concerns about potential limitations. We agree that is it practically very difficult to enforce a matrix to be strictly random following a particular distribution (such as Gaussian or Uniform). We use constructions from the torch library such as torch.nn.init.kaiming_uniform to generate pseudo-random matrices which is close to the best one can expect to achieve given that true randomness is unattainable in computational environments.

• Q2: How can we guarantee there are no training instabilities for larger scale models ?

Regarding potential instabilities, our experiments scaling up to 8B parameters with LLama3 have not revealed any training instabilities. Could you kindly elaborate on the potential instability risks associated with larger models the reviewer is referring to ?

• Q3: How does the approximation error proposed in Theorem 4.1 behave with model size ?

As model size increases, the number of values to estimate in $W$ grow quadratically, while trainable parameters in $\Lambda$ and $\Gamma$ increase linearly (similarly to LoRA or VeRA for example). All else being equal, this could theoretically suggests increasing approximation errors with larger models. We however point out that experiments with architectures up to 8B LLama3 show that this phenomenon is absent and that RandLoRA maintains performance.

An important detail invalidating this intuition is that $W$ is a fine-tuned update on top of pre-trained weights $W_0$ that tend to become more expressive with increased model size and thus requiring simpler weight updates [1,2]. This would intuitively explain why $\epsilon$ (equation 6) remains small in practice. We observe this phenomenon in CLIP experiments, where VeRA's competitiveness improves with larger ViT-H/14 models (Figure 3). This derivation together with experimental results of Table 1 support RandLoRA's suitability for large model applications.

In the hypothetical scenario where an accuracy degradations occurs, users can adjust the slice size r to estimate smaller, more manageable portions of W, thus reducing $\epsilon$. This adaptability parallels LoRA's adjustment of the rank r for harder problems.

• Q4: Do sparse matrices preserve the full rank constraint ?

The main concern for ternary sparse matrices could come from drawing the same row in the B matrices more than once over $N$ tries which would lead to a co-linearity and thus non-full rank. If we can show that the probability of this even happening is reasonably small then the full rank constraint would be preserved in practice.

Given we draw $N$ matrices of size $D\times d$, this problem simplifies to $N\times D$ tries at drawing the same or exact opposite sequence of length $d$. Using the probability assignments from L480-481, we calculate that the probability of two rows being the same or opposite is $2\times (\frac{1}{2})^{d}= 1\times 10^{-231}$ for the smallest matrix we train on: ViT-B-32 of size $768\times 768$. Given N = 100, a fair estimate of a common configuration for RandLoRA, the probability of having any same or opposite sequence over $(N\times D)$ tries is $2\times (N\times D)(N\times D - 1)(\frac{1}{2})^{d}= 8\times 10^{-222}$ which enables sparse matrices to preserve the full rank in practice.

We have added this clarifying comment at the end of section 6.4.

[1] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." ICLR 2022.

[2] Aghajanyan, Armen, Luke Zettlemoyer, and Sonal Gupta. "Intrinsic dimensionality explains the effectiveness of language model fine-tuning." ACL 2021.

• Q5: Full fine-tuning as a baseline for language experiments.

A Full-finetuning (FT) is included for as a baseline for vision and vision-language experiments in Figures 2, 3 and Tables 8-11 in the supplementary material. We omitted this baseline for the language experiments due to VRAM constraints on LLama3-8b particularly.

We however report here a baseline for the smaller Phi3 network on the commonsense reasoning task where we find that full fine-tuning performs worse than parameter-efficient methods especially as the amount of training data increases. This most likely a result of over-fitting.

We expect the same behavior for larger architectures such as LLama3-8b and refer to GPT-3 results in the original LoRA paper where LoRA performs better than full finetuning for these larger models (Table 4 in the original LoRA paper [1]). In conclusion, we use LoRA as our baseline because it performs better than full fine-tuning for large language models.

• Q6: Are the larger improvements on Vision language due to increased complexity ?

Yes we concur with the analysis of the reviewer on the increase in complexity of the task, probably as well due to the dual backbone nature of the vision-language architectures. We added a line at the end of Section 5.3 using this insight to justify the improvements on Vision-language tasks.

• Q7: More ablations on sparse random bases and type of random base distributions deserve further study.

We provide early results on the effect of sparse bases in Table 3 where we observe a small decrease in accuracy results over the widely used CLIP and LLama3 architectures. We expect these results to generalize to other architectures as well.

Regarding the ablations on the distribution the random bases are drawn from, we refer the reviewer the VeRA paper which performed such studies (Table 6 and 7 in the VeRA paper). Because the VeRA formulation is currently the base for our sliced SVD approximation (equation 6), we used VeRA's design choices regarding random base creation and expect their ablation results to hold for RandLoRA.

Since VeRA did not propose to use sparse bases however, we are now working on providing ablation experiments using random bases with different levels of sparsity [3] as well as Gaussian bases to add to Table 4. Early results indicate slight decreases of accuracy with sparser bases and Gaussian bases. We will report the full results as soon as they become available.

• Q8: Run RandLoRA on LLama-70B and LLava-35B.

We acknowledge and share your interest in exploring RandLoRA's performance on extreme model sizes. Although we haven't reported results on these very large architectures, Figure 3 and Table 1 demonstrate a promising trend: RandLoRA consistently improves upon LoRA as network size increases, suggesting scalability to even larger architectures.

We have attempted to train a LoRA baseline on a quantized version of LLama-70B on 4xA100 GPUs and calculated it would require a 6 days runtime, inference and hyper-parameter search excluded. Given the two-week response period and our limited compute resources, fine-tuning these architectures in time is unfortunately not realizable.

[1] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." ICLR 2022.

[3] Li, Ping, Trevor J. Hastie, and Kenneth W. Church. "Very sparse random projections." ACM SIGKDD international conference on Knowledge discovery and data mining. 2006.

We thank the reviewer again for their detailed review. Please let us know if you have further concerns you would like addressed.

Dear reviewer rgkh,

We have now obtained the results for the suggest ablation study of the random bases. We have studied normally distributed and uniform binary random bases as well as sparser variations for the sparse bases. We find that by following the maximum sparsity suggestion of Li et al [3] we obtain 93% sparse bases that maintain close performance to the dense random bases. When pushing above this limit to 98% or 99% sparsity we observe significant degradations in performance.

We have added these results to Table 4 and discussed them line 524-530 in Section 6.4 of the updated manuscript (blue text). We are additionally working on evaluating the 93% sparse bases for LLama3 and we report the results soon.

The authors

Dear Reviewer,

Could you kindly respond and indicate whether authors have addressed your concerns?

Thanks, AC

We thank the reviewer for the detailed review of our work.

• Q1: What is the difference between VeRA and RandLoRA ?

A discussion is available in Section 6.5 which we detail further here.

RandLoRA and VeRA stem from distinct hypotheses regarding the optimal rank of weight updates when fine-tuning large pre-trained model. VeRA adopts LoRA's low-rank approximation, while RandLoRA targets full-rank updates. Essentially, VeRA efficiently approximates LoRA, whereas RandLoRA explicitly identifies non-full ranks as the limit to the quality of the update over the amount of trainable parameters.

For context, RandLoRA sums over all rank-r sections of the full-rank weight update's SVD decomposition (Section 4.2) by leveraging parameter-efficient estimations of each section. Similarities with VeRA here stem from the empirical confirmation that VeRA's formulation is effective at estimating these low-rank sections. In this context, RandLoRA's framework allows for future improvements to SVD section approximations, such as other formulations different to VeRA's that further reduce in Equation 6 or by using hybrid estimations such as explicit estimation of critical sections using LoRA (lines 558-565).

We further clarify here that RandLoRA's current solution is not simply a sum of multiple VeRAs since: we propose the use of a single random base A to constrain memory use (equation 4), demonstrate the applicability of sparse random bases to reduce compute requirements (section 6.4) and derive theoretical bounds to justify convergence (Theorem 4.1). Finally VeRA is capped by definition at (D+r) trainable parameters for weight matrices whereas RandLoRA accommodates for larger budgets (line 233).

To conclude, VeRA and RandLoRA cater to different needs of the parameter-efficient community. While VeRA enhances LoRA's parameter efficiency for ultra-small parameter budgets, RandLoRA adresses tasks requireing moderately larger budgets (Figure 1).

• Q2 GLUE and E2E results for RandLoRA

We appreciate your suggestion to include GLUE and E2E results for comprehensive comparison which we are now working on providing. In the interim, we refer the reviewer to Section 5.4 and Table 1, showcasing RandLoRA's enhancements over the LoRA and VeRA baselines in challenging commonsense reasoning language tasks. Notably, RandLoRA demonstrates good improvements with the LLama3-8B architecture which is highly relevant to the research community.

Regarding GLUE, we anticipate modest improvements of RandLoRA over LoRA due to three factors: 1) GLUE tasks primarily involve simple binary classification, 2) LoRA already bridges the gap with full fine-tuning, and 3) VeRA reports performances on par with LoRA's.

We have provided results for GLUE using the RoBERTa-base architecture (125M parameters), averaged over 5 runs. As expected, the results for each algorithm are very close and not statistically different. We will make the results for the RoBERTa-large architecture available as soon as they come in with early trends pointing to similar results to the RoBERTa-base architecture. We are additionally currently running RandLoRA on E2E with GPT2-medium.

Results on GLUE datasets with RoBERTa-base. We report the same metrics as in the VeRA paper: Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for the remaining tasks. As VeRA did not release their code, we use hugginface's transformers official implementation: https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification . Contrary to VeRA and to save on large search times, we use the same hyper-parameters across all tasks. All algorithms are run using the same code base.

Thank you again for your review, we will share further results here as they become available. Please let us know if you have remaining concerns you would like addressed.

Dear reviewer ufJr,

Please find below further results. We have run the RoBERTa-large architecture on GLUE using the same settings as specified in our previous comment (5 runs).

We find that in this case, RandLoRA improves over both LoRA and VeRA. These results indicate how RandLoRA becomes more beneficial for larger network sizes. This observation is in line with the results presented in Figure 3 where RandLoRA outperforms fine-tuning more with increased network sizes.

We additionally report here results on the E2E dataset with GPT2-Medium, we use LoRA's official implementation and the recommended hyper-parameters. We find that RandLoRA slightly outperforms LoRA for the BLEU and NIST scores. We are still working on reproducing VeRA's results.

The authors

Thank you for the rebuttal. Most of my concerns are solved. I'm willing to raise my score.

We thank the reviewer for the detailed review of our work.

• Q1: Why does RandLoRA outperform fine-tuning in various tasks like image classification?

RandLoRA indeed improves performance over fine-tuning in some cases, especially for vision-and-language models such as CLIP, as shown in Figure 3 where RandLoRA outperforms fine-tuning despite having fewer trainable parameters. We find that in these cases, full fine-tuning is highly susceptible to over-fitting, thus compromising generalization accuracy. In contrast, RandLoRA optimizes over a lower-dimensional parameter space, which mitigates over-fitting and leads to better generalization in this case. This is corroborated by the observation that RandLoRA outperforms fine-tuning more on larger models in Figure 3.

The superior performance of low-rank methods when fine-tuning language models is also observed in the original paper, where LoRA [1] is reported to sometimes outperform full fine-tuning. This is particularly evident in the GPT-3 experiments (Table 4 in the LoRA paper [1]). We also evidence this phenomenon when full-finetuning Phi3 on the commonsense reasoning tasks. This is further detailed in the response to reviewer rgkh.

To shed light on the overfitting, we would like to point out that in Figure 5 in the supplementary material, particularly in Figure 5b, full fine-tuning leads to much deeper training loss minima. These results highlight how that lowered sensitivity to over-fitting is the main reason behind RandLoRA's improvements over full-finetuning in CLIP or language architectures.

• Q2: Given equal training parameters, why is it better to perform a full rank update.

Full rank updates allow to explore any direction in the weight space. This is in contrast to LoRA which restricts exploring $r$ (rank) directions. In some cases such as vision-language (Figure 3) or commonsense reasoning (Table 1) we find that having access to all directions in the weight space become important to train accurate models and as a result RandLoRA outperforms LoRA.

• Q3: RandLoRA should compare with prompt-tuning baselines which are specifically designed for few-shot image classification.

Thank you for suggesting these related works, we have now included them in the new Section 2.3 to better contextualize our work in the parameter-efficient landscape.

Although they tackle similar problems, we argue that these algorithms are orthogonal to our proposed method. Hence, RandLoRA should not be considered in direct competition but complementary. Probably for the same reason, the papers suggested by the reviewer also do not compare with LoRA. In general, while prompt-tuning is designed for few-shot settings exclusively, RandLoRA's competitiveness increases with moderately larger parameter budgets (Figure 1) and generalizes to full dataset fine-tuning.

A comprehensive investigation into the performance of LoRA compared to prompt-tuning algorithms has however been recently been performed [2] where LoRA was found to be comparable to recent prompt-tuning alternatives for low-shot scenarios but largely outperformed for larger amounts of shots. This discussion has been summarized and included in Section 2.3 of the paper. We agree that further research demonstrating the complementarity of both approaches would be relevant to the community but it is currently beyond the scope of this work.

We are however working now on running the PromptSRC+DePT configuration as an extension to our vision experiments to better contextualize our work in the parameter-efficient landscape and confirm the results of [2]. We will post the results here if they complete before the rebuttal deadline and include them in the camera ready.

[1] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." ICLR 2022.

[2] Zanella, Maxime, and Ismail Ben Ayed. "Low-Rank Few-Shot Adaptation of Vision-Language Models." CVPR 2024.

Thank you again for your review, please let us know if you have remaining concerns you would like addressed.

Dear reviewer uANM,

Prompt tuning experiments have now completed.

We have been unable to run the PromptSRC+DePT configuration as the code for this configuration is not publicly released. The DePT paper reports Maple[3]+DePT as a close second best so we have chosen to report this configuration instead as it accurately reflect state-of-the-art performances for prompt tuning.

We report results for 4 and 16-shots over the datasets selected in the DePT paper. We train CLIP in its ViT-B/32 variant with Maple[3]+DePT, RandLoRA-10 and LoRA-16 all training approximately 3M parameters.

We find, as suggested in our previous comment, that while the Maple+Dept configuration compares favorably for the 4-shots setting, it struggles to keep up for 16-shots. We additionally report that Maple + DePT requires a much longer training time than both LoRA and RandLoRA, especially as the number of classes increases. For example, training 16-shots for 10 epochs on ImageNet requires 3.5h and 17GB of VRAM for Maple + DePT while it requires 2 minutes and 4.5GB of VRAM for RandLoRA.

The authors

[3] Khattak, Muhammad Uzair, et al. "Maple: Multi-modal prompt learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Dear Reviewer,

Could you kindly respond and indicate whether authors have addressed your concerns?

Thanks, AC

Thank the authors for addressing most of my concerns. I will raise my score.

Manuscript and supplementary material update

We have updated the manuscript with further results according to the reviewer's suggestions.

Results for GLUE and E2E benchmarks can be found in Table 5 in appendix B.1 and B.2 respectively. Results for Maple+DePT as a state-of-the-art prompt-tuning baseline is available in Table 6 in appendix B.3. We have referenced these additional results in the main manuscript lines 287-289.

We have updated the ablation study of Table 4 with additional results using up to 99% sparse bases. We are now working on applying the 93% sparse bases to LLama3 to complement the 66% sparse results that were initially available.

Results for Table 2 comparing RandLoRA with DoRA are now available. We have used DoRA's official implementation and found that for commonsense reasoning, DoRA performs very similarly to LoRA. Furthermore we now report the training time for DoRA as well.

We find that because DoRA's normalization strategy necessitates to explicitly compute the $BA$ update, it leads to a 2.2x increase in training time over LoRA with no obvious improvement paths. As a comparison, RandLoRA-30 leads to a 1.7x increase. We have included DoRA's training time for LLama3 in appendix C.6.1.

Only reviewer ZGK2 answered so far

As the requested additional experiment have now completed, we invite the reviewers to reassess our work in light of these additional results and of our responses to their concerns. If their concerns have been addressed, we encourage the reviewers to share their revised evaluation and to consider increasing their acceptance score.

The authors