RidgeLoRA: Matrix Ridge Enhanced Low-Rank Adaptation of Large Language Models

We sincerely appreciate Reviewer PKfP's insightful review. Our point-by-point responses are provided as follows.

The paper acknowledges the absence of scaling experiments due to resource limitations, which slightly limits conclusions on its applicability to frontier LLMs.

We appreciate that reviewers have pointed the scaling experiments out. Indeed, experiments with larger models are absolutely crucial. We have now included additional scaling experiments with Llama-2-13b to address this concern. The results further validate our method’s effectiveness on larger, frontier LLMs.

From this table, it can be seen that as the model parameter scale increases, RidgeLoRA consistently outperforms other baselines, with an average improvement of 4% over vanilla LoRA.

Can the authors provide further empirical justification for why this change leads to improved representation and learning dynamics? For example, include further analysis or ablation comparing intermediate representations (e.g., gradients) between series and parallel designs to explain the observed performance gains.

We thank Reviewer PKfP for this good suggestion that can definitely provide more evidence for the improved training efficiency. We tracked the gradient norm (grad_norm) over training steps for RidgeLoRA, PiSSA, KaSA, DoRA and vanilla LoRA. RidgeLoRA and PiSSA showcase higher gradient norms than other LoRA variants, indicating stronger learning dynamics. This supports that RidgeLoRA improves representation and training efficiency. We will add the curve of grad_norm from different variants to the revised version.

We sincerely appreciate Reviewer S5wb's insightful comments and suggestions. Our point-by-point responses are provided below.

The current RidgeLoRA algorithm is a multiplicative update to W while LoRA is an additive update to W. It is not an apples-to-apples to comparison between the rank of a additive and the rank of a multiplicative update. So apart from the trainable ridge term \Sigma, has the authors considered comparing the effect of additive VS multiplicative updates by setting \Sigma = I?

Thank you for this suggestion. To better isolate the effect of the multiplicative structure from the trainable ridge term $\Sigma$, we conducted an additional experiment where we kept the ridge values frozen as the identity matrix (i.e., $\Sigma = I$). This effectively removes the contribution of the trainable ridge term while preserving the multiplicative nature of the update.

We observe that although the performance of this is slightly weaker than the full RidgeLoRA with a learnable $\Sigma$, it still significantly outperforms standard additive LoRA and most of its known variants. This result suggests that the multiplicative structure itself contributes meaningfully to performance improvements, even without additional trainable parameters from $\Sigma$. In other words, the benefit of RidgeLoRA is not solely due to parameter count or flexibility, but also due to the structural advantage of multiplicative adaptation.

I also find the use of the terms serial and parallel as illustrated in Figure 1 slightly confusing. It seems to me much more natural to describe the updates as additive or multiplicative to the weight matrix W.

Thank you for the suggestion. We agree that describing the updates as additive or multiplicative to the weight matrix W may be more intuitive than using the terms parallel and serial. In the revision, we will revise the terminology and clarify the expressions—i.e., $X(W + AB)$ (additive) and $X(\lambda \Sigma + AB)W$ (multiplicative)—to improve clarity and readability.

Theorem 3.1 is related to the approximation property of adding a ridge term to matrices in general, and does not seem to have any direct relevance to finetuning performance. We all know that adding more parameters or increasing the rank should decrease the approximation error, so the theorem doesn't really contain any special insight on why the algorithm works.

We thank Reviewer S5wb for the thoughtful comment. While Theorem 3.1 primarily addresses the approximation error, this quantity is tightly connected to the model's ability to fit the parameter updates during finetuning. In particular, a lower approximation error indicates that the modified weight matrix more faithfully captures the intended update directions. Theorem 3.1 thus provides a theoretical foundation for why adding the ridge term leads to better representational fidelity. Rather than being a generic observation that more parameters help, the theorem quantitatively shows how the ridge term specifically improves the approximation in a non-trivial manner, helping explain the empirical gains we observe.

Why could RidgeLoRA use fewer parameters than LoRA for Llama-2-7B in Table 2 and Table 3 when it has extra diagonal parameters?

We thank Reviewer S5wb for carefully examining our work, here we further clarify the details of RidgeLoRA to avoid misunderstandings. As illustrated in Figure 1, the reason RidgeLoRA can use fewer parameters than LoRA despite introducing an additional diagonal term lies in the structural difference between the two methods.

While vanilla LoRA adopts a parallel connection (or additive) structure of the form $X(W+AB)$, RidgeLoRA employs a series connection (or multiplicative) formalized as $X(\lambda\Sigma + AB)W$. This formulation not only introduces the ridge term $\Sigma$, but also changes the shape of matrix B from $d_{out}\times r$ to $d_{in} \times r$, as reflected in the right part of Figure 1.

Because of this change, the number of trainable parameters in $B$ is reduced. As a result, even with the added diagonal matrix $\Sigma$, which is rather lightweight compared with $B$, RidgeLoRA maintains a lower overall parameter size. Comparisons on the parameter size can also be found in the second column of Table 1.

The use of ridge term also bears some resemblance to the method IA^3, which makes use of multiplicative diagonal scaling of the weight matrices as trainable parameters. The authors can consider including it in the discussion of related works.

We sincerely thank Reviewer S5wb for pointing this out. IA^3[1] introduces trainable multiplicative diagonal scaling to adapt pre-trained models. As another PEFT method that is different from LoRA and its variants, IA^3 bears conceptual resemblance to RidgeLoRA’s use of the ridge term. We will include a discussion of IA^3 in the related work section and add the appropriate citation in the revised version.

[1] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel.2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems 35 (2022), 1950–1965.

We appreciate the review of Reviewer WvKZ for its constructive suggestions. Our detailed rebuttal is as follows.

The nuclear norm plot in Figure 2 appears to suggest that the weight updates still remain relatively low rank (similar to that of PiSSA) on GQA.

Analyzing rank behavior across tasks with varying intrinsic dimensionality is indeed insightful. In response, we performed additional analysis using checkpoints from a diverse set of tasks (model weights trained on MetaMathQA, CommonSense, and Code-Feedback) and found that RidgeLoRA consistently yields higher nuclear norms compared to other LoRA variants. This suggests that RidgeLoRA is more capable of adapting to tasks that require higher-rank updates.

Due to the constraints of the rebuttal format (e.g., no figures or external links), we are unable to include the full visualizations and detailed comparisons here. However, we will provide comprehensive nuclear norm plots and cross-task analysis in the final version of the paper.

It is not completely obvious to me whether the series connection necessarily has an advantage over the parallel approach in terms of predictive performance. The improvements observed in Table 6 seem very marginal. Perhaps the claim about the necessity of the serial connection requires additional validation.

Here we include results as instructed by Reviewer S5wb to prove that the multiplicative structure (or serial connection) itself contributes a lot to the good performance.

To better isolate the effect of the multiplicative structure from the trainable ridge term $\Sigma$, we conducted an additional experiment where we kept the ridge values frozen as the identity matrix (i.e., $\Sigma = I$). This effectively removes the contribution of the trainable ridge term while preserving the multiplicative nature of the update.

We observe that although the performance of this is slightly weaker than the full RidgeLoRA with a learnable $\Sigma$, it still significantly outperforms standard additive LoRA and most of its known variants. This result suggests that the multiplicative structure itself contributes meaningfully to performance improvements, even without additional trainable parameters from $\Sigma$. In other words, the benefit of RidgeLoRA is not solely due to parameter count or flexibility, but also due to the structural advantage of multiplicative adaptation.

Could the authors provide additional explanation as to why the numbers for Llama-3.1-8B and Mistral-v0.3-7B in Table 2 are often much worse than all of the low-rank methods? I am aware that the paper mentions this is similar to the trends observed in other papers, but am wondering if adding sufficient regularization can update the numbers to be more reflective of true full fine-tuning performance.

As discussed in the paper, similar trends have been reported in prior work (e.g., DoRA, PiSSA, and KaSA), particularly in instruction tuning settings, where full fine-tuning (FFT) can be prone to overfitting or instability without careful regularization. In this context, LoRA can be seen as a form of regularization that constrains the parameter update space, which may help explain why it sometimes outperforms unconstrained FFT. Our work focuses specifically on the low-rank setups, and comprehensive results showcase that RidgeLoRA outperforms other LoRA variants.

In our experiments, we adopted commonly used FFT configurations without extensive hyper-parameter tuning. One key reason for this choice is fairness: although our primary focus is on comparing low-rank adaptation methods, we include FFT (training without regularization on weight updates) as a reference. To ensure meaningful comparisons, all methods within the same category (FFT or low-rank) are trained under consistent hyper-parameters. That is, all low-rank methods, including RidgeLoRA, share the same setup, and all FFT experiments follow a fixed default configuration.

We greatly appreciate Reviewer WvKZ’s feedback on our rebuttal and are pleased to share the detailed nuclear norm results of weight updates across different datasets. Figure 2 from the original submission is on CommonSense dataset, while the others are presented as follows:

From these results, we observe that RidgeLoRA consistently achieves significantly higher nuclear norms than other LoRA variants (PiSSA, KaSA, vanilla LoRA), indicating a greater capacity to induce higher-rank updates. This trend holds robustly across multiple tasks, supporting our claim that RidgeLoRA is better suited for tasks requiring richer adaptation compared to standard LoRA methods. We will also properly discuss our observations together with those from [1] in the revised version.

Also, for concern W1 and W2, as instructed by other reviewers, here we provide results from Llama-2-13B to address concerns w.r.t. other model scales.

Evaluations are limited to the 7-8B model scale, and it is possible that the approximation quality and thus the gap vs. the full fine-tuning baseline shows different trends at other model scales.

Performance improvements are relatively small vs. other low-rank baselines with more recent LLMs (Llama-3.1-8B and Mistral-v0.3-7B).

As noted, the relative gains on newer LLMs (Llama-3.1-8B, Mistral-v0.3-7B) are smaller compared to older models. This is expected, as more recent base models are trained with improved synthesized data and carefully designed instruction-tuning recipes, leaving less headroom for further improvement by alternative low-rank variants.

Nevertheless, RidgeLoRA still delivers consistent gains in these settings without introducing additional parameters or computational overhead. These improvements remain practically meaningful and can be readily applied to other domains that require low-rank adaptations.

[1] Biderman et al., LoRA Learns Less and Forgets Less, Transactions on Machine Learning Research, 2024.

We thank Reviewer pZr9 for the professional review. Our rebuttal is as follows.

Could the authors provide a more detailed analysis of RidgeLoRA’s performance on models larger than 8B parameters?

We appreciate that reviewers have pointed the scaling experiments out. Indeed, experiments with larger models are absolutely crucial. We have now included additional scaling experiments with Llama-2-13b to address this concern. The results further validate our method’s effectiveness on larger, frontier LLMs.

From this table, it can be seen that as the model parameter scale increases, RidgeLoRA consistently outperforms other baselines, with an average improvement of 4% over vanilla LoRA.

Could the authors provide case studies to illustrate RidgeLoRA’s superior performance?

Thank you for the suggestion. Here we provide an illustrative case demonstrating RidgeLoRA’s superior reasoning capability compared to vanilla LoRA tuning.

Consider the following question:

Question: A rooster lays an egg on the top of a slanted roof. Which way does the egg roll?
A. To the left, because the roof is slanted that way.
B. To the right, since the slope leads downward.
C. It stays on the roof because it's balanced.
D. It doesn’t roll at all—roosters don’t lay eggs.

The Llama model fine-tuned with vanilla LoRA answers:

The answer is B. To the right, since the slope leads downward. 

In contrast, the RidgeLoRA-enhanced Llama model correctly identifies the trick and answers:

The answer is D. It doesn’t roll at all—roosters don’t lay eggs. Roosters don’t lay eggs, so this is a trick question.

This example shows that RidgeLoRA can better capture nuanced reasoning and common-sense knowledge. We will include more case studies in other domains such as math and code problems in the revised version to further illustrate RidgeLoRA’s advantages.