Uni-LoRA: One Vector is All You Need

Dear Reviewer 5yKv,

Weakness 1 – Limited baselines in Tables 3 and 4

Thanks for the suggestion. Tables 3 and 4 initially included only 2–3 baselines because we aimed to include the results reported in their original papers. For Table 3, we have now added two additional baselines: VeRA and FourierFT. When reproducing these baselines, we used their original hyperparameter configurations for fine-tuning.

The results demonstrate that Uni-LoRA achieves comparable performance to VeRA on mathematical reasoning tasks, while requiring only half as many trainable parameters on Mistral-7B and less than one-third on Gemma-7B. As of FourierFT, Uni-LoRA consistently outperforms it across both tasks, while using fewer trainable parameters. Furthermore, Uni-LoRA requires only 15 hours of training time under the same hardware setup, which is significantly faster than FourierFT (73 hours), and comparable to VeRA.

As for Table 4, the experimental setup is based on QLoRA-quantized models. Unfortunately, as stated in their documentation or code comments, the publicly released implementations of LoRA-XS and FourierFT are currently not compatible with quantized base models, so we could not include them as baselines in this setting.

Weakness 2 – Inference efficiency

Thanks for the question. Similar to LoRA, at inference time, Uni-LoRA doesn’t introduce any memory and compute overhead beyond the backbone models since B and A matrices are merged to the backbone models. The difference between LoRA and Uni-LoRA is that after training, Uni-LoRA regenerates B and A matrices through $P×θ_d$, and then merges B and A to the backbone models for inference.

Weakness 3, Question 3 – GPU memory consumption

Thanks for the question. Similar to prior works such as VeRA, VB-LoRA, FourierFT, Tied-LoRA, and LoRA-XS, our focus is on reducing the number of trainable parameters of LoRA. We do not claim reductions in memory or training cost compared to LoRA, and we only claim that Uni-LoRA reduces the storage cost of LoRA parameters significantly. Reducing the number of trainable parameters can substantially decrease both storage and transmission cost for model deployment and is highly valuable in industry. For example, in federated learning (FL), it can significantly lower communication costs while preserving user privacy. We will clarify this further in the manuscript.

Question 1 – Training stability, particularly given the random initialization

Thanks for the question. In Table 2 of the main paper, we report the median and standard deviation across five different seeds for each method on six tasks. Below, we present the average of the standard deviations across these six tasks:

As can be seen, our method demonstrates comparable stability to other baselines. For RoBERTa-Base, only VeRA has a slightly lower average standard deviation than Uni-LoRA. For RoBERTa-Large, both VeRA and FourierFT have slightly better stability, but Uni-LoRA remains highly competitive in terms of stability. We will clarify this in the main paper.

Question 2 – Results under varying values of r and d

Thanks for the suggestion. We have now added the ablation on r and d. Please refer to our responses to Reviewer FicU, Disadvantage 1, Question 3 and Reviewer 6tip, Weakness 5.

Question 4 – Uni-LoRA on reasoning-focused models

Thanks for the suggestion. We apologize that we couldn’t provide preliminary results on reasoning-focused models given our limited experience on reasoning models and a very tight rebuttal schedule. However, we could provide some insights in this area.

To the best of our knowledge, several recent works have successfully applied LoRA to reasoning-focused models [1-3]. Since Uni-LoRA essentially reconstructs the LoRA parameters from a low parameter space, we believe it can in principle be applied to any models that LoRA performs well (including NLP and CV models, and see our response to Reviewer FicU on Uni-LoRA for CV tasks). It’s interesting to extend Uni-LoRA to the reasoning-focused models, and we plan to explore this direction in the future.

References:

[1] Li, Xiang, et al. "MoDE-CoTD: Chain-of-Thought Distillation for Complex Reasoning Tasks with Mixture of Decoupled LoRA-Experts." LREC-COLING 2024.
[2] Wang, Shangshang, et al. "Tina: Tiny reasoning models via lora." arXiv preprint arXiv:2504.15777 (2025).
[3] Xue, Yihao, and Baharan Mirzasoleiman. "LoRA is All You Need for Safety Alignment of Reasoning LLMs." arXiv preprint arXiv:2507.17075 (2025).

Dear Reviewer k62c,

Weakness 1, Question 1 – On the Importance of Isometry

Thanks for the question. We ensure that the projection matrix P in Uni-LoRA is isometric for the following key reasons:
Yes, standard gradient descent methods are not parameterization-invariant. Poorly scaled parameterizations, which distort the geometric structure of the optimization landscape, can lead to unstable step sizes and slow convergence. Non-isometric transformation can make changes sensitive in one direction and insensitive in another direction. (Please see: J. Nocedal and S. J. Wright, Numerical Optimization 2nd ed., Chapters 2.2 and 4.5 for more discussions on the effects of poor scaling.)

Although heuristics such as gradient clipping and gradient-normalized Adam [1] exist as remedies, we prefer to maintain isometry as a desirable property from the outset in reparameterization. This ensures that the conditioning of the optimization problem remains unchanged -- no artificial ill-conditioning is introduced by the reparameterization. We will add this discussion in the updated manuscript.

References:
[1] Yu, Adams Wei, et al. "Block-normalized gradient method: An empirical study for training deep neural network." arXiv preprint arXiv:1707.04822 (2017).

Minor Issues

• Unable to find the code for reproducing some experiments (Fastfood)
  For reproducibility, we will open-source all our Uni-LoRA code, including our Fastfood implementation and the Uni-LoRA code for ViT fine-tuning introduced during the rebuttal.

• LoRA-XS could be described more clearly: SVD is not mentioned.
  We apologize for the lack of clarity regarding LoRA-XS. We will clarify that its LoRA-like matrices A and B are derived from the SVD of the pretrained weight matrix W.

• L57: Consider clarifying in which way is theoretically grounded.
  Thanks for pointing this out. As we analyzed in Sec. 3.3, P possesses three desired properties for adaptation performance, i.e., globality, uniformity/load-balanced, and isometry. We will clarify this at L57 and add a reference to Sec. 3.3.

• Figure 1: "top-k of learnable logits" is mentioned in the legend, but I don't see it being used.
  Thanks for pointing this out. The “top-k of learnable logits” is specific for VB-LoRA, and refers to the orange boxes in the VB-LoRA illustration (Fig. 1d). In this case, k=2 refers to the two diagonal sub-matrices. We will clarify this in Fig. 1.

• L167: I don't find it clear why "Apparently" is used.
  Thanks for the question. Since the projection matrix P in standard LoRA is just an identity matrix, which satisfies PᵀP = I, it is inherently isometric.

Question 2 – Any intuition why LoRA outperforms other methods on MATH, but not GSM8k?

Thanks for pointing this out. This is an interesting observation that we neglected before. We don't have a solid explanation or intuition at this time. We will investigate this in our future work.

Thank you for the clarifications and additional experiments! Having considered all reviews and responses to them, I see no important weakness. Therefore, I will keep my positive assessment of the work.

Dear Reviewer 6tip,

Weakness 1 – Analysis of Method Stability

Thanks for the question. In Table 2 of the main paper, we report the median and standard deviation across five different seeds for each method on six tasks. Below, we present the average of the standard deviations across these six tasks:

As can be seen, our method demonstrates comparable stability to other baselines. For RoBERTa-Base, only VeRA has a slightly lower average standard deviation than Uni-LoRA. For RoBERTa-Large, both VeRA and Tied-LoRA have slightly better stability, but Uni-LoRA remains highly competitive in terms of stability. We will clarify this in the main paper.

Weakness 2 – Clarification on Theorem 1

We would like to clarify that Theorem 1 assumes $n_j > 0$, indicating that each column of the projection matrix P must have at least one non-zero entry (see Theorem 1). The counter example provided does not satisfy this assumption.

In practice, the chance to sample a P in which one column is entirely zero is extremely low because P is a very tall and narrow matrix (where each row has only one non-zero element at a random location). For example, in the LLaMA-2 7B experiments in Sec. 4.3, P has a shape of 10M × 0.52M. In this case, the probability that a column is entirely zero is less than 0.23%.

To ensure the theorem’s condition ($n_j > 0$) is always held, we can re-sample P if one column happens to be all zeros. We will clarify this further in the main paper.

Weakness 3 – LoRA vs. FFT and Impact of Reducing the Number of LoRA Parameters

Numerous studies have shown that LoRA significantly reduces memory and compute cost while achieving comparable or even superior performance to full fine-tuning (FFT) [1-5]. Its efficiency and effectiveness have led to widespread adoption in industry.

Meanwhile, the parameter redundancy of LoRA has been explored in several works, such as VeRA, Tied-LoRA, VB-LoRA, and LoRA-XS. These methods reduce the number of trainable parameters while maintaining similar performance to LoRA, highlighting the parameter redundancy of LoRA. Uni-LoRA builds upon this insight and further improves the parameter efficiency for LoRA.

References:
[1] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." ICLR 1.2 (2022): 3.
[2] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." NeurIPS 36 (2023): 10088-10115.
[3] Biderman, Dan, et al. "LoRA Learns Less and Forgets Less." Transactions on Machine Learning Research.
[4] Shuttleworth, Reece, et al. "Lora vs full fine-tuning: An illusion of equivalence." arXiv preprint arXiv:2410.21228 (2024).
[5] Lian, Chenyu, et al. "Less could be better: Parameter-efficient fine-tuning advances medical vision foundation models." arXiv:2401.12215 (2024).

Weakness 4 – Why Does Local Projection Perform Worse Than Global Projection?

Thanks for the question. As highlighted by AdaLoRA [1], when fine-tuning with LoRA, not all layers contribute equally, i.e., the importance and information content of each LoRA module varies. Prior methods such as VeRA, Tied-LoRA, and LoRA-XS use local projections with the same projection dimension for each layer, which means layers with different information content are projected into different subspaces of the same size. This same-sized treatment for each layer is suboptimal. In contrast, global projection projects all LoRA parameters into a shared subspace to mitigate the aforementioned issue.

Local projection is analogous to pouring five unequal volumes of water into five identical cups, which may lead to overflow in some cups and under-utilize others. In contrast, global projection is analogous to pouring all water into a single container, which allows more efficient utilization of capacity. The ablation studies in Table 6 support this claim. We will clarify this further in the updated manuscript.

References:
[1] Zhang, Qingru, et al. "Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning." ICLR 2023.

Weakness 5 – Additional Analysis on the Influence of Rank r

Thanks for the suggestion. For a fair comparison, in the paper we followed VB-LoRA and set the LoRA rank r = 4. To evaluate the influence of rank r, here we conducted additional analysis by varying r for RoBERTa-Large on the SST-2 task (from GLUE) and Gemma-7B on the mathematical reasoning benchmarks. The results reported in the table below show that Uni-LoRA maintains stable performance across a wide range of ranks, with r = 4 achieving the best balance between accuracy and efficiency. We will include this ablation study in the updated manuscript.

SST-2 Accuracy vs. r

GSM8K / MATH Accuracy vs. r

Weakness 6 – formatting issue

Thanks for pointing this out. We will fix it in the updated manuscript.

Dear Reviewer FicU，

Disadvantage 1, Question 3 – Impact of Projection Dimension d on Model Performance

Thanks for the suggestion. First, we would like to clarify that our projection matrix P is not optimized via backpropagation. Instead, P is randomly sampled and fixed (see Theorem 1). Since P is not learned, our method incurs lower memory and computational overhead compared to the baseline methods, such as VBLoRA and Tied-LoRA, which require optimizing projection parameters.

To investigate the impact of d on model performance, we conducted ablation studies by fine-tuning RoBERTa-Large on the SST-2 task (from GLUE) and Gemma-7B on the mathematical reasoning benchmarks, where d is varied while keeping all other settings at their default values. The results are summarized in the tables below:

SST-2 Accuracy vs. d

GSM8K / MATH Accuracy vs. d

Our experiments show that performance improves rapidly with increasing d when d is small, and then plateaus as d grows larger. We will include these results in the updated manuscript.

Disadvantage 2, Question 2, Limitation 1 – Performance on More Complex Tasks

To evaluate Uni-LoRA in more complex tasks, we followed the FourierFT setup for computer vision and conducted additional experiments with the results reported in the table below.

Specifically, we applied Uni-LoRA with d = 74,000 on ViT-Base and d = 144,000 on ViT-Large, for rank r = 4. A grid search was conducted over:

• head learning rate: {1e-3, 2e-3, 5e-3, 1e-2}
• trainable vector $\theta_d$ learning rate: {2e-3, 5e-3, 1e-2, 2e-2, 5e-2}

while fixing the number of training epochs to 10.

Each experiment was repeated five times, and we reported the mean and standard deviation across the five runs.

All results, except those for Uni-LoRA, are taken from the original FourierFT paper, including Full Fine-tuning (FF) and Linear Probing (LP), in which only the classification head is fine-tuned.

Experiments show that given the same number of trainable parameters Uni-LoRA significantly outperforms FourierFT across computer vision benchmarks. Moreover, when the number of parameters in FourierFT is three times that of Uni-LoRA, their performances on computer vision benchmarks are comparable. These results demonstrate the strong generalizability of Uni-LoRA to complex tasks.

Our core observation is that tasks (NLP or CV) that benefit from LoRA also tend to perform well with Uni-LoRA. This is because Uni-LoRA essentially reconstructs the LoRA parameters from a low parameter space via $P\times\theta_d$. We will incorporate these CV results into the main paper and release all the code for reproducibility.

Regarding multiple projection matrices, this is a very interesting idea, and we plan to explore it in our future work.

Question1 – Consider geodesic distance on manifolds to optimize P?

Thank you for the suggestion. First of all, as we discussed above, the projection matrix P is not optimized. Instead, P is randomly sampled and fixed (see Theorem 1). Since P is not learned, the number of trainable and stored parameters of Uni-LoRA is extremely small (i.e., $|\theta_d|$ + 1 random seed). Optimizing P would increase the number of trainable and stored parameters dramatically and incur large memory and computational overhead.

If we ignore the memory and computational overhead and optimize P in the non-Euclidean space, it might be possible to replace Adam with its Riemannian adaptive counterparts [1, 2]. Adapting to the manifold setting, however, requires computing the function’s gradient and, in some cases, the Hessian matrix -- which is computationally prohibitive. The practicality of methods and libraries such as Geoopt [2] and Pymanopt [3] for large-scale deep learning optimization remains uncertain.

References:
[1] Bonnabel, Silvere. "Stochastic gradient descent on Riemannian manifolds." IEEE Transactions on Automatic Control 58.9 (2013): 2217-2229.
[2] Bécigneul, Gary, and Octavian-Eugen Ganea. "Riemannian Adaptive Optimization Methods." International Conference on Learning Representations (ICLR 2019). Vol. 9. Curran, 2023.
[3] Townsend, James, Niklas Koep, and Sebastian Weichwald. "Pymanopt: A python toolbox for optimization on manifolds using automatic differentiation." Journal of Machine Learning Research 17.137 (2016): 1-5.

Limitation 2 – Scalability in Distributed Systems

Thank you for the suggestion. We apologize that we were unable to fine-tune on GPT-3 (175B) due to resource constraints. However, Uni-LoRA naturally supports data parallelism, and indeed, the results reported in Table 3 were obtained using multi-GPU data-parallel training.

To enable model parallel training of Uni-LoRA across multiple GPUs, we can adopt a strategy where each GPU maintains a local copy of the trainable vector $\theta_d$, and synchronization is achieved via an all-reduce operation on the gradients during backpropagation. This approach is highly efficient for Uni-LoRA, as the number of trainable parameters is extremely small, resulting in negligible communication overhead across devices.

We would like to emphasize that the extremely small number of trainable parameters constitutes a key advantage of Uni-LoRA over other PEFT methods. For example, when fine-tuning LLaMA2-13B, Uni-LoRA requires synchronizing only 1M parameters, whereas VB-LoRA may require synchronization of up to 256M parameters in the worst case. We will include this discussion in the updated manuscript.