LoRA vs Full Fine-tuning: An Illusion of Equivalence

We thank you for your thoughtful review and are grateful for your positivity about our work. We have incorporated the suggestions you have requested, including typos, presentation changes, and updating claims, into the updated manuscript. We provide responses and more context where required below:

There are a few claims that are not entirely supported: line 166: "LoRA introduces new singular vectors that have a large contribution to the norm of the updated parameter matrix." - As far as I can see there is no support for a change in norm, only in directions because of using cosine similarity.

Thank you for pointing this out. While Figure 5 shows an increase in the singular value corresponding to the intruder dimension—suggesting a potential increase in the matrix norm—we currently don't report the norm itself. We will include this statistic in future revisions to more concretely support the claim.

Different distance measures Currently, the authors only consider cosine similarity as distance measures between singular vectors. It would be interesting if intruder dimensions could be identified via different distance measures, or whether it is really only the difference in directions that enables identifying them.

Thank you for the suggestion; it's a valuable point. While we focus on cosine similarity, we don’t claim it's the only viable measure. We chose it because, in the context of SVD, comparing singular vectors via cosine similarity offers a natural and interpretable way to track changes in the principal directions of the weight matrix. Since these directions define how the matrix transforms input, shifts in them are meaningful. That said, exploring alternative distance measures is an interesting direction for future work and could offer complementary insights.

Another very interesting point would be showing where exactly the intruder dimensions appear in. Are they only apparent in certain projection matrices (Q/K/V, etc.) and layers and can they be related with implicit biases as introduced in [4]?

Across our experiments, intruders appear in all weight matrices, with no systematic difference in the number of intruder dimensions between MLP and attention matrices. We will add in a section outlining these details to the appendix.

It would be interesting how the number of intruder dimensions changes with larger models.

We thank the reviewer for there extensive suggestions of additional experiments. Due to the limited length of the rebuttal, we will leave these experiments to future work to extend this work.

We thank the reviewer for their thoughtful review. We provide responses to their review below.

The core concern is that the paper primarily presents extensive empirical comparisons between LoRA and full fine-tuning to illustrate the emergence and impact of intruder dimensions, without proposing new methods or actionable improvements based on these insights. As per the NeurIPS regulation, it may be better suited for the Datasets & Benchmarks track rather than the main conference.

We are confused by your suggestion to change tracks. First, under the main track call for papers, the website states that NeurIPS "encourage in-depth analysis of existing methods that provide new insights in terms of their limitations or behaviour beyond the scope of the original work". Moreover, this paper provides neither a dataset nor a benchmark. Further, we do indeed provide actionable insights: we show several ways that the number of intruder dimensions can be reduced, including with certain initializations (Fig. 10), lower learning rates (Fig. 7), and setting $\alpha=2r$ (Section N). We even identify a simple post-training intervention motivated by section 5: scaling down the magnitude of intruder dimensions can reduce the forgetting a model has while retaining nearly identical adaptation performance. These are concrete prescriptions that can be used to train better LoRA models that exhibit reduced forgetting and better out-of-distribution generalization.

Despite the extensive empirical evidence for the impact of intruder dimension, the theoretical explanation for why they appear in fine-tuning LoRA is still lacking.

We do provide a few mathematical justifications for the occurrence of intruder dimensions in Section B.1. We showed that adding the outer product of a random vector to a weight matrix introduces an intruder dimension (Section B.1). This is analogous to the matrix product of BA in LoRA when rank is 1, and implies that intruder dimensions occur because the B and A vectors are uncorrelated to the columns/rows of the weight matrix W_0. Further, we showed empirically that just training the B matrix, while freezing the A matrix with all singular values of 1, reduces the amplification of the singular values because of the matrix product and therefore reduces the number of high ranking intruder dimensions, aligning with our mathematical intuition.

Mathematically, it is expected that constrained updates (e.g., via LoRA) tend to concentrate on directions associated with the largest singular values, while full fine-tuning distributes updates more evenly with sufficiently large update space. Without rigorous theoretical support, the introduction of the intruder dimension appears to be a restatement of this known behavior within the specific framing of LoRA.

What you describe as expected behavior actually doesn’t align with what our observations. Rather than changes occurring primarily along the existing high-ranking singular vectors, what happens is that a few new high-ranking singular vectors emerge, while the original ones remain largely unchanged or only slightly shifted. This suggests that the dynamics introduced by constrained updates like LoRA are more nuanced than simply concentrating on pre-existing dominant directions like you describe. Whereas full fine-tuning distributes updates across all singular vectors.

It is important to note that, before our work, it was not expected that LoRA would update a weight matrix in a structurally different manner than full fine-tuning. Rather, It was expected that LoRA approximated full fine-tuning (https://arxiv.org/pdf/2106.09685), which would result in the LoRA looking like an approximation of the fully fine-tuned weight updates that is concentrated on the top existing singular values, like you mention.

Regarding the correlation between the number of intruder dimension and catastrophic forgetting (Lines 256-263), the experiments using a larger learning rate show that LoRA is not well optimized, as evidenced by a significant performance gap. In such cases, the increase in intruder dimensions could be a byproduct of suboptimal training rather than an inherent issue. For a well-optimized LoRA model (e.g., with lr=1e-4), the observation in [1] (LoRA learns less but forgets less) still holds. Thus, conclusions based on poorly trained models are less convincing.

We are concerned the reviewer is misinterpretting this section. All the models are equally well trained (as measured by equal validation and train loss) on the adaptation distribution--which are typical measures used when finetuning models. Could you please explain which measure would make one of the models sub-optimal over the other?

In the section you refer to, we sweep learning rates to see its effect on optimization. We find that we get similar test loss but significantly different forgetting profiles. For example, in Fig. 7, let us examine LoRA with lr=1e-4 and 2e-4. We see that the models have trained to within 0.5% test accuracy performance across epochs (middle plot). However, we see that 2e-4 consistently forgets a much more than 1e-4 when examining their corresponding pseudo loss. These are the datapoints that are used to indicate that intruder dimensions correlate with forgetting. We infact support the conclusion of ("LoRA learns less but forgets less") and extend it in this paper (lines 235-247).

Our main conclusions of our paper are all based on well trained models. We welcome the reviewer to follow up and provide clarification if we have misunderstood their concern. If there is another criterion by which one model is considered suboptimal and is typically monitored at finetuning, we would appreciate further clarification.

In Appendix H.2, the observation that performance drops when learned knowledge is perturbed and LoRA's preserved knowledge is exaggerated is unsurprising. This behavior is consistent with general deep learning dynamics and does not convincingly support the claim that intruder dimensions degrade OOD performance. The argument feels more like an expected consequence of disrupting well-internalized features.

Your suggested intuition is actually opposite to our findings, making this a surprising result. Our finding is not that "performance drops when learned knowledge is perturbed", but rather that it is barely changed when intruder dimensions are scaled down while forgetting has decreased significantly. We provide a key passage in our main text (lines 290-295) here:

"In one example for LLaMA2-7B fine-tuned on MetaMath with LoRA r = 256, we observe that scaling the top intruder dimension in each matrix with $\lambda = 0.3$ leads to a 0.1% drop in test accuracy and a 33.3% drop in the forgetting induced by fine-tuning. In another for RoBERTa-base fine-tuned on QQP, using $\lambda = 0.7$ leads to equivalent in test accuracy and a 33.2% reduction in the forgetting induced by fine-tuning. In certain scenarios, we even see test accuracy improve along with a drop in forgetting. If we instead increase their contribution ($\lambda > 1$), we observe more forgetting."

This is an unexpected result. We welcome the reviewer to follow up and provide clarification if we have misunderstood their concern.

In Appendix I, was the hyperparameter search of learning rate conducted to ensure that LoRA with different alpha converged to similar test accuracy? If not, it's trivial that improper/insufficient learning naturally leads to low rank solutions.

Yes. We follow standard machine learning practice while conduct learning rate sweeps and ensure that the different models compared converged to similar test accuracy as shown in Table 2. In Appendix I, all models train to similar test loss and accuracy, making our investigation fair across ranks.

We thank the reviewer for their thoughtful review. We provide responses to their review below.

Logical mismatch in Section 5. Section 4 convincingly correlates the number of intruder dimensions with forgetting, yet Section 5 only experiments with scaling down the magnitude of the top intruder dimension. To solidify the argument, the authors should also manipulate the count of intruder dimensions and measure its effect on forgetting.

We actually do study altering the count of intruder dimensions in section 5. When scaling with $\lambda=0$, we remove that intruder dimension. In Section 5, we remove the top intruder dimension in each weight matrix. This has the effect of removing up to 72 intruder dimensions from a RoBERTa model and has the effect of significantly reducing forgetting while causing less extreme drop in test performance. For examples, see Fig. 8 when $\lambda=0$. We hope this clarifies the reviewers confusion.

Limited practical guidance. The paper does not provide guidance on how to prevent the emergence of intruder dimensions during training, which is crucial for improving real-world applicability. As it stands, intruder dimensions appear to be primarily useful as a post-hoc model selection criterion.

This paper provides several concrete suggestions on how to reduce the emergence of intruder dimensions at training time. We provide evidence that certain initializations (Fig. 10), lower learning rates (Fig. 7), and setting $\alpha=2r$ (Section N) all lead to fewer intruder dimensions. These are concrete prescriptions that can be used to prevent the emergence of intruder dimensions during training.

Moreover, we show that scaling down the top intruder dimensions after LoRA fine-tuning recovers the drop in pretraining performance without affecting downstream adaptation. In one such case, we see forgetting reduced by 33.2% with no change in test accuracy (lines 292-293). This effectively resurfaces forgotten information without compromising task performance—a novel and actionable intervention that leads to improved OOD generalization which was not previously known.

These are a few practical guidelines offered by the paper.

The paper concludes that intruder dimensions cause forgetting. However, full-tuning with few intruder dimensions, still exhibits more forgetting than LoRA with an appropriately chose alpha. How do we understand this based on findings in the paper?

It is important to note that intruder dimensions are a cause of forgetting in LoRA'd models, but not the only cause of forgetting in fine-tuning in general. This is obvious, since we should expect any sort of deviation from the pre-trained weights, which were trained to minimize language modelling loss, to lead to an increase in language modelling loss (aka forgetting). This finding demonstrates that these two methods update weight matrices in fundamentally different ways. This explains the structural basis of forgetting with LoRA'd models (and variants of LoRA). However, a functional explanation of forgetting in full-finetuned models can be different and an interesting avenue for future study.

In the continual learning setup, are the similarity matrices in Figure 9 all computed with the original model? I am wondering can we observe intruder dimensions over a model already tuned on another task(say do weights after task 3 have intruder dimension over weights after task 2)?

Yes, all similarity matrices in Figure 9 are computed using the original model. That said, we do observe what you're describing: as we train on more tasks, new intruder dimensions emerge in distinct locations with every round of adaptation to a new task (highlighted in pink), indicating that intruder dimensions can indeed arise relative to earlier task weights (e.g., after Task 3 vs. Task 2).

Thank you for your thoughtful review. We have incorporated your suggestions, including the mistakes you caught like VeRA having the same line twice in Fig. 23, into our updated manuscript.

Where required, we provide responses to your review below:

Main weaknesses: The focus of the paper. The paper focuses on intruder dimensions as the main factor distinguishing the spectral properties of LoRA and full fine-tuning, but it is not entirely clear whether this choice captures the core difference.

Our primary objective was to assess whether different fine-tuning methods converged to functionally equivalent models despite differing parameterizations. To investigate this, we perform a spectral analysis to characterize how the sequence of transformations each method applied differ.

Since the SVD breaks up the principal components of a matrix, studying how these principal components change (via cosine similarity) is an intuitive way to examine how a weight matrix is changed during fine-tuning. Empirically, we observe that while full fine-tuning, which directly manipulates the weights, makes small adjustments to the magnitude and direction of the singular vectors, LoRA, which uses a low rank product, introduces new singular vectors with low cosine similarity to the existing singular vectors. We provide some intuition for why intruder dimensions are the right framework in Section B.2, where we showed that adding the outer product of a random vector to a weight matrix introduces an intruder dimension. This is analogous to the matrix product of BA in LoRA when rank is 1, and implies that intruder dimensions occur because the B and A vectors are uncorrelated to the columns/rows of the weight matrix W_0.

We have conducted an extremely detailed examination and would have every reason to report a better measure had it been found. Does the reviewer have a suggestion of measure?

The paper does not provide a clear explanation for why LoRA introduces intruder dimensions. Moreover, while the results convincingly show that LoRA solutions have intruder dimensions, the causal relationship is not fully established.

It is important to note that intruder dimensions are an empirical observation of LoRA. However, we provide a few mathematical justifications for their occurrence. We showed that adding the outer product of a random vector to a weight matrix introduces an intruder dimension. This is analogous to the matrix product of BA in LoRA when rank is 1, and implies that intruder dimensions occur because the B and A vectors are uncorrelated to the columns/rows of the weight matrix W_0 (Section B.2). Further, we showed empirically that just training the B matrix, while freezing the A matrix with all singular values of 1, eliminates the amplification of the singular values because of the matrix product and therefore reduces the number of high ranking intruder dimensions. This suggests that this multiplicative property may cause new singular vectors to have large singular value. These experiments provide evidence towards why LoRA causes intruder dimensions to be introduced.

it is possible that these dimensions are not specific to LoRA, but instead arise from larger overall changes to the weights ... The experiments varying the learning rate for LoRA show that higher learning rates lead to more intruder dimensions, and it is plausible that full fine-tuning would show a similar pattern if higher learning rates were used. This raises the question of whether the observed differences between LoRA and full fine-tuning are due to specific properties of the LoRA method, or simply a consequence of differences in optimal hyperparameter ranges.

The reviewer is absolutely correct to be curious about this and we indeed investigated this. In our investigation, we used standard learning rates for both full fine-tuning and LoRA. When we conduct a large learning rate sweep for both methods, we observe that increasing learning rate significantly over the default setting leads to training instabilities and divergence. Decreasing learning rate has difficulty converging to similar performance, even with more training steps. Because of this, the resulting models perform significantly worse and therefore cannot be used in our analysis. For models that do converge to similar, near optimal performance (Fig. 7), we observe that full fine-tuning contains no intruder dimensions while LoRA contains many.

Practical importance of intruder dimensions. The analyses in Sections 4 and 5 do not clearly establish a causal link between intruder dimensions and forgetting or generalization.

In Section 4, we find a strong correlation between intruder dimensions and forgetting. The reviewer is absolutely right to point out that this could be the result of a third variable that is causing both to increase. To show that this is not the case, in section 5 we intervene on the intruder dimensions. In it, we scale down the singular values of the intruder dimensions, which has the effect of reducing their contribution on the fine-tuned weights. By performing this causal experiment, we find that there is little change in test accuracy but a large impact on forgetting. One example of this (reported in the main text in lines 292-293) shows that using $\lambda=0.7$ on our model trained on QQP leads to no change in test accuracy but a 33.2% reduction in forgetting.

These findings lead to the following conclusion: intervening on intruder dimensions and scaling them down results in a big drop in forgetting but little change in test accuracy, showing that these intruder dimensions, and in particular their magnitude (singular vector), causes a large amount of the forgetting in LoRA models.

We do not claim that intruder dimensions are not the only possible cause of forgetting. Any sort of change to the pre-trained weights should be expected to impact a models base language modelling ability. Indeed, we observe that full fine-tuning makes small adjustments to the direction and magnitude of the pre-trained singular vectors. This still changes the fine-tuned weights, and we should expect forgetting to occur. Our causal experiment scaling intruder dimensions is specific to LoRA.

Intruder dimensions and section 5 explain the structural basis of forgetting with LoRA'd models (and variants of LoRA). However, a functional explanation of forgetting in full-finetuned models can be different and an interesting avenue for future study.

Additional concerns, comments and questions: The claim that LoRA forgets less than full fine-tuning, even at comparable performance levels, is not particularly strong. The results for LoRA and full fine-tuning in Table 2 differ significantly, and the observed differences in forgetting appear to correlate, at least to some extent, with differences in accuracy. For example, on the QQP task, both accuracy and forgetting differ between high-rank LoRA and full fine-tuning, while on MNLI, the results are almost identical in both respects.

Our conclusion that "LoRA forgets less than full fine-tuning, even at comparable performance levels" stems from our results in Table 2 and Fig. 6. Table 2 contains the test accuracies of all of our models. In it, we see models trained to approximately the same accuracy. We are not sure why the reviewer claims that the results for LoRA and full fine-tuning in Table 2 differ significantly. For example, in Table 2, all our models fine-tuned on MNLI perform within 0.5% of each other. Looking horizontally, comparing full fine-tuning and LoRA r=16, without loss of generality, we see similar performance, with LoRA sometimes outperforming full fine-tuning and vice versa. Further, when correlating test accuracy and forgetting within datasets for Table 2 (as indicated by the reviewer), we get no statistically significant result (test: Spearman's rank-order correlation, p-value>0.33 for each.). Therefore, this means that observed differences in forgetting do not correlate with differences in accuracy.

When looking at Fig. 6b, we see that full fine-tuning always forgets more than LoRA. This leads us to conclude that LoRA forgets less than full fine-tuning, even at comparable performance levels, which exends the findings of [1]. We hope this clarifies the reviewers concern.

[1] - https://arxiv.org/pdf/2405.09673

A more detailed analysis of intruder dimensions of different LoRA variants would be beneficial. There appears to be a non-monotonic dependence of the number of intruder dimensions on the epsilon parameter, which likely reflects properties of specific methods used.

We provide a preliminary study to show that our findings scale to other variants in Section P. We find that LoRA variants (LoRA+, VeRA, AdaLoRA, PiSSA) all have intruder dimensions, showing that these methods are not adequate for preventing intruder dimensions. We emphasize that our study of LoRA variants is orthogonal to our main study, and that it is difficult to conduct every detailed experiment we have on all LoRA variants. While we save an in depth analysis for future work, we have no reason to suspect these will impact the claims we make in this paper.