One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation

Thank you for your feedback. We address the raised weaknesses as follows.

Novelty:

We respectfully disagree with the reviewer on the novelty aspect. Incremental SVD has not been used on activations before which brings its own sets of problems. PiSSA [1] applies SVD to pre-trained weights and AdaLoRA [2] does not use SVD, but merely SVD-like re-formulation of LoRA, which is not the same. Furthermore, neither [3] nor [4] actually apply SVD to activations. [3] applies a dot product between activations and weights to obtain importance scores and [4] relies on scaling salient activation regions. In fact, neither of [3,4] have a single mention of SVD in their work.

Applying SVD to inputs to weight matrices (we refer to them as activation vectors for simplicity) requires dealing with growing memory as the data matrix increases with each batch, therefore we employ incremental SVD to keep memory costs constant. We clarified this in our updated method section and also explicitly mention our contributions which are as follows: We propose a novel data-driven initialization scheme for LoRA by leveraging incremental SVD on minibatches of activation vectors. We propose a data-driven heuristic for adaptive rank allocation based on explained variance. We demonstrate EVA's effectiveness across a variety of different domains and tasks.

We also added this list of contributions in our introduction in the revised version

Right-singular vectors capture directions of most variance:

We provide a proof in Appendix H which demonstrates that the right-singular values obtained via SVD are equivalent to the projection onto the principal components which is usually obtained via PCA. In fact, practical implementations of PCA (pytorch, scikit-learn, etc.) usually employ SVD under the hood to obtain the projection onto the principal components. The intuition for leveraging this projection as initialization for the matrix A in LoRA is that the projection onto the principal components capture the most variance, i.e. they preserve the most information of the downstream task in a linear lower dimensional subspace. By redistributing the ranks we can maximize the amount of variance that is explained throughout the model. This leads to improved performance while even reducing the amount of trainable parameters (see Table 11 in Appendix B3). For convenience we also added this table below.

References:

[1] Pissa: Principal singular values and singular vectors adaptation of large language models., Meng, et al. NeurIPS 2024

[2] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, Zhang et al., ICLR 2023

[3] A simple and effective pruning approach for large language models., Sun et al., arXiv preprint arXiv:2306.11695 (2023).

[4] AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration., Lin et al., PMLR 2024

Dear reviewer EQQt,

We thank you again for taking the time and effort to help improve our paper and believe we have addressed all your concerns by clarifying our contributions, improving clarity and adding supplementary proofs and more mathematical formulations.

We would be grateful for an opportunity to discuss wether there are any pending concerns you can point us to.

Thank you, Authors

Thank you for taking the time to clarify your concerns.

Novelty:

Firstly, we would like to emphasize that the novel idea of our work is to make initialization data-driven. This has not been considered in prior work. To achieve such an initialization, we perform incremental SVD on activations. Using SVD as a tool on activations is just the means to achieve the data-driven initialization and does not diminish the idea of data-driven initialization. It is not an extension to [1] as the core idea of using the downstream data is fundamentally different to using information from the pre-trained weights. We would also like to stress again that [2] does not use SVD. Finally, we do not just use SVD, but incremental SVD, which brings its own set of challenges, such as dealing with incremental updates and distributing them across the entire large model in an efficient manner. To clarify, our contributions are as follows:

• We propose a novel data-driven initialization scheme for LoRA by leveraging incremental SVD on minibatches of activation vectors.
• We propose a data-driven heuristic for adaptive rank allocation based on explained variance.
• We demonstrate the effectiveness of EVA across a variety of different domains.

Rationale behind EVA:

Depending on the downstream data, each layer’s activation will exhibit different patterns as they are sensitive to the task and triggered to highlight different aspects of the pre-trained weight. Therefore, the activations capture task context. Our aim is to leverage this context for adapting the FM. In fact, we hypothesize that the more information we can leverage from this context, the better the downstream performance, i.e. we require a projection that propagates the maximum amount of information, which can be obtained by a projection onto the principal components, i.e. via SVD. We verified empirically in our experiments that EVA consistently leads to lower loss and improved performance. We are currently also running another experiment where we use the components that capture the least amount of variance and will post the performances as soon as they are finished. This experiment should verify that the directions capturing the most variance should be used for initialization.

References:

[1] Pissa: Principal singular values and singular vectors adaptation of large language models., Meng, et al. NeurIPS 2024

[2] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, Zhang et al., ICLR 2023

Thank you for your further engagement and the increase in score.

We have now further clarified the motivation in a more formal manner and related the effect of our initilization to the fine-tuning performance in the revised version. Let $X = U \Sigma V^\top$ be the SVD decomposition on activations $X$. Then initializing $A = V_{:r}$ and $B=V_{:r}^\top$ minimizes the reconstruction error $\lVert X - XV_{:r}V_{:r}^\top \rVert$. This fact is given by the Eckart-young theorem [1] as $U_{:r} \Sigma_{:r} V_{:r}^\top$ is the best reconstruction of $X$. Now by choosing $A=V_{:r}$ the downprojection $XA$ must contain the most information about $X$ according to the data processing inequality [2], as the maximum amount of information $B$ can contribute is $B=V_{:r}^\top$. Now the gradient w.r.t. $B$ is $\frac{\partial \mathcal{L}}{ \partial B} = \frac{\partial \mathcal{L}}{\partial W}A^\top$. The fine-tuning process is concerned with storing information about the data in the weights. By choosing $A = V_{:r}$ we guarantee that the maximum amount of information is available at the beginning of training, such that it only needs to be learned what information to keep, i.e. what parts of $XA$ are relevant for the downstream task.

[1] The approximation of one matrix by another of lower rank. Eckart et al., Psychometrika 1936

[2] An intuitive proof of the data processing inequality, Beaudry et al., Quantum Information & Computation 2012

Finally, we would like to recite our presented table from our previous answer, where it is clearly visible that while improvements are marginal for some tasks, EVA also reduces the amount of trainable parameters (see Table 11 in Appendix B3). The reason for this is that more ranks are transferred from the higher dimensional feedforward weights (non-attention weights) to the attention weights. We again present this table below. It clearly shows that EVA is pareto-dominant compared to all competitors.

We would like to thank the reviewer for their constructive feedback and address the raised weaknesses and questions as follows.

Presentation:

We agree that Figure 1 did not accurately reflect the procedure conducted by EVA and would like to thank the reviewer for bringing this to our attention. We updated Figure 1 in the revised manuscript. We also added the mathematical formulation of the incremental update in Section 3.2 along with pseudocode in Algorithm 2 and references to proofs plus a supporting proof in Appendix H that the components obtained via incremental SVD are projections onto the principal components that explain the most amount of variance in the linear subspace spanned by EVA. The changes are highlighted in red color in the revised version. If the reviewer has any further suggestions to enhance clarity, we will gladly update our manuscript.

Training Convergence:

Thank you for the suggestion to add more training curves to our manuscript. We added additional loss curves for Gemma-2, Llama-3.1, and Llama-2 in Figure 4 in the Appendix. They verify the results shown in Figure 2. We will add further loss curves for the different domains as well, but need to re-train some models for that. In Figure 2 on the right, we show the gradient norm at the very first logging step, which demonstrates that EVA receives the strongest gradient signal at the beginning of training.

Naming of the method:

The reason why we name our method “Explained Variance Adaptation” is because the projection onto the linear subspace which we use to initialize the matrix A explains the most variance of the input to each weight matrix. This is supported by the proof in Appendix H that we added which shows that the right singular values after SVD are a projection onto the principal components, i.e. they explain the most variance. Further, by adaptively allocating ranks we can maximize the explained variance throughout the model. The reason why this results in better performance is that EVA projects the input to a weight matrix into a lower dimensional subspace while preserving the most amount of information possible. Therefore, EVA propagates more information of the downstream data through the model compared to random or a weight-driven initialization.

Ablation Studies:

This is indeed an interesting question and the motivation to perform these experiments on the VTAB-1K dataset was to gain a better intuition on what factors are crucial for performance on in vs out-of-distribution data. We found that by varying the scale and the directions of our initialization there is variation in the downstream behavior. For example, using whitening (changing the scale) leads to improvements on structured images. Therefore, this setting may be especially useful when dealing with out-of-distribution data. Further, EVA is particularly effective on natural images. Therefore, by changing the scale in a controlled manner we can make EVA more suitable for out of distribution data. Furthermore, varying the directions of the projection matrices deteriorates performance on out-of-distribution data. Therefore, we can conclude that the combination of both scale and direction of the projection matrix are crucial for good performance of EVA. We moved this section to appendix to accommodate a more in-depth description of EVA in Section 3, however still added another paragraph to make this more explicit.

Dear reviewer iw7g,

We thank you again for taking the time and effort to help improve our paper and believe we have addressed all your concerns by properly motivating EVA, updating Figure 1 and extending the methodology section, supplementary proofs and additional pseudocode, adding additional loss curves, and clearing up any confusion about the ablation studies.

We would be grateful for an opportunity to discuss wether there are any pending concerns you can point us to.

Thank you, Authors

Thank you for taking the time to respond to our rebuttal!

To clarify, we did not claim that Appendix H addresses any rationale about EVA, but it addresses concerns raised by reviewer EQQt that the claim about explained variance is unsupported - it is not, it is proven in Appendix H. We explicitly give a more intuitive rationale on EVA below.

Rationale behind EVA:

Depending on the downstream data, each layer’s activation will exhibit different patterns as they are sensitive to the task and triggered to highlight different aspects of the pre-trained weight. Therefore, the activations capture task context. Our aim is to leverage this context for adapting the FM. In fact, we hypothesize that the more information we can leverage from this context, the better the downstream performance, i.e. we require a projection that propagates the maximum amount of information, which can be obtained by a projection onto the principal components via SVD. We verified empirically in our experiments that EVA consistently leads to lower loss and improved performance.

Faster convergence:

We provided evidence for faster convergence for three different models on three different downstream tasks. Can the reviewer elaborate on how many more loss curves it takes to be convincing?

Effect on initialization:

There is plenty of evidence in the literature that initialization of LoRA significantly impacts downstream performance, may it be of empirical [1,2,3] or analytical nature [4]. Even the original work on LoRA [5] conducted pre-training on different tasks as initialization for certain other downstream tasks as it improved performance. Can the reviewer provide any references that provide evidence that initialization methods have "minor impact"?

References:

[1] Pissa: Principal singular values and singular vectors adaptation of large language models., Meng, et al. NeurIPS 2024

[2] LoRA-GA: Low-Rank Adaptation with Gradient Approximation, Wang et al., NeurIPS 2024

[3] OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models, Büyükakyüz et al., arXiv:2406.01775

[4] The Impact of Initialization on LoRA Finetuning Dynamics, Hayou et al., NeurIPS 2024

[5] LoRA: Low-Rank Adaptation of Large Language Models, Hu et al., ICLR 2022

Motivation behind EVA:

LoRA initializes the matrix B with zeros and matrix A at random. This has beneficial properties (such as performing well under high learning rates [1]), but leads to a weak and noisy gradient signal at the initial training stage [2]. To obtain a better training signal we initialize the matrix A with the projection onto the principal components, i.e. the directions that explain the most variance. This has the effect that the most information possible is propagated through the linear subspace. The projection can be obtained by performing SVD on inputs to weight matrices (we refer to them as activation vectors for simplicity). Since we need to handle arbitrarily large datasets it is not a good solution to collect and store activations and perform SVD on the full datamatrix. Therefore, we perform SVD incrementally and stop after we do not observe any change in the projection onto the principal components anymore. We clarified this point in the revised version and added the relation to PCA (proof in Appendix H) along with formulation of the incremental SVD (Section 3.2) accompanied by pseudocode (Algorithm 2) and references to supplementary proofs.

Presentation:

Thank you for pointing out the inconsistency in the figures. We updated the fontsize of the left part of Figure 2. Furthermore, we updated Figure 1 to clarify the initialization procedure performed by EVA. It should be a lot more clear now. If the reviewer has any further suggestions for enhancing clarity we will gladly include them into our manuscript.

Significance of results:

There is compelling evidence in the literature that proper initialization of the LoRA subspace results in significant improvements [2,3,4]. While the differences may appear small at first glance, it is important to keep in mind that a small difference in average performance requires improvement in several single tasks. As EVA consistently improves upon its competitors on average across all domains, our reported results suggest statistical significance. Furthermore, we added a table that shows that after rank redistribution EVA actually reduces the number of trainable parameters while simultaneously improving performance. In the table below we illustrate that EVA improves upon LoRA for the same rank budget while reducing the parameter count.

Computational Overhead of EVA:

We disagree that the additional computational overhead diminishes the effect of LoRA fine-tuning. In fact, we added Table 22 in the appendix that demonstrates that initialization with EVA merely constitutes 0.2% of the total training time (in minutes), i.e. initialization takes approx. 70 seconds compared to 8 hours of training for Llama-2-7B on the common sense reasoning tasks. We also show a reduced form of the table below for convenience.

In this table LoRA-GA is concurrent work on data-driven initialization which we added to our experiments in Table 2 and 3 in the revised version of our manuscript. The only other initialization method that is faster than EVA is OLoRA which is weight-driven and performs worse in our experiments. Therefore, EVA attains a better performance vs complexity trade-off. Finally, as demonstrated above, EVA additionally reduces the parameter count via rank redistribution which further reduces computational complexity. Considering these facts, we believe EVA provides significant contributions to the community.

References:

[1] The Impact of Initialization on LoRA Finetuning Dynamics, Hayou et al., NeurIPS 2024

[2] Pissa: Principal singular values and singular vectors adaptation of large language models., Meng, et al. NeurIPS 2024

[3] LoRA-GA: Low-Rank Adaptation with Gradient Approximation, Wang et al., NeurIPS 2024

[4] OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models, Büyükakyüz et al., arXiv:2406.01775

Dear reviewer Qji4,

We thank you again for taking the time and effort to help improve our paper and believe we have addressed all your concerns by properly motivating EVA, updating Figure 1, and clearing any doubts of significance of results, since EVA reduces the number of trainable parameters, and providing evidence that overhead induces from EVA is negligible.

We would be grateful for an opportunity to discuss wether there are any pending concerns you can point us to.

Thank you, Authors

Lack of baselines:

Thank you for making us aware of the concurrent work on LoRA-GA, we incorporated it into our experiments and added results for it in Table 2 and 3. We find that EVA mostly outperforms LoRA-GA, see reduced table below.

Furthermore, we analyzed the efficiency of LoRA-GA in Table 22 in Appendix F4 and show that initialization with EVA is approximately 12 times faster for Llama-2-7B on MetaMath. We again show a reduced version of this table below.

rsLoRA and LoRA+ are orthogonal approaches as they focus on rank-stabilized scaling and learning rate optimization. Suggested modifications by either one of them can be applied to ALL methods including EVA. To investigate their effect, we added an experiment where we compare rsLoRA and LoRA+ with EVA under the same conditions to fine-tuning LLama-2-7B on the common sense reasoning tasks. We find that (1) alpha=1 performs better in our experiment setup than alpha=32 (83.4 vs 82.7 avg perfomance, respectively), (2) rsLoRA improves LoRA as well as EVA, (3) LoRA+ does not lead to any gains in our setup, (4) EVA consistently outperforms LoRA across all these settings. We provide the table below.

We would like to thank the reviewer for the time spent to provide constructive feedback that helps us significantly improve our manuscript. We adress all raised weaknesses as follows.

Rationale behind EVA:

Thank you for pointing out the lack of clarity! The rationale behind EVA is that we would like to initialize the matrix A in a way such that the most information of the downstream task is being propagated through the model. The reasoning behind this is that random initialization introduces wasteful update steps especially in the beginning of training [1]. We can remediate this by initializing A with a projection onto the principal components, because those explain the most amount of variance in the data. In our case the data is the input to each weight matrix, which we refer to as activation vectors for simplicity. An efficient way of obtaining this projection is via SVD (we added a proof in Appendix H for equivalence between right-singular vectors and projection onto principal components). A naive way would be to collect activations for the entire dataset and perform SVD on the stacked matrix. However since the dataset can be arbitrarily large and we want to avoid any overhead in memory we leverage incremental SVD [2]. We added the mathematical formulation of the incremental procedure and references to proofs in Section 3.2 along with pseudocode for the incremental update in Algorithm 2 in the Appendix. Now, to maximize the amount of variance that is explained throughout the model , we perform rank redistribution by sorting and assigning ranks according to their explained variance. To enhance clarity, we also updated Figure 1 such that it comprises all parts of EVA.

Parameter budget and adaptive ranks:

Thank you for raising this issue! For $\rho=1$ the parameter count is exactly the same as for LoRA, however this changes for $\rho=2$. We added a table containing the number of trainable parameters and performance for the larger models (Table 11 in Appendix B.3) which demonstrates that EVA with rank redistribution actually reduces the number of trainable parameters while improving performance. We added a reduced version of this table below for convenience.

The reason for this is that ranks are often transferred from the higher dimensional feed-forward weights to lower dimensional attention weights. Furthermore, the reviewer is correct in highlighting that adaptive rank allocation is not always the best performing method and this heavily depends on the downstream task, i.e. on common sense reasoning $\rho=2$ performs better and on math tasks $\rho=1$ performs slightly better. Further, for the image classification experiments and the language understanding experiments we used adaptive ranks, while for the RL tasks uniform ranks performed better. We have made this more explicit in our revised version. We also added an additional paragraph in the discussion section where we elaborate on the effect of rank redistribution in more detail. We would like to stress that there is no one method that performs best across all tasks, i.e. no free lunch, but rank redistribution can improve EVA while simultaneously reducing the parameter count. Therefore we believe it is a valuable contribution.

References:

[1] Pissa: Principal singular values and singular vectors adaptation of large language models., Meng, et al. NeurIPS 2024

[2] Incremental learning for robust visual tracking., Ross et al., NeurIPS 2004

See next response for remaining points.

Dear reviewer nwRB,

We thank you again for taking the time and effort to help improve our paper and believe we have addressed all your concerns by properly motivating EVA, demonstrating that EVA actually reduces the number of trainable parameters compared to LoRA, and by providing comparisons to works such as rsLoRA, LoRA+, LoRA-GA and different values for $\alpha$.

We would be grateful for an opportunity to discuss wether there are any pending concerns you can point us to.

Thank you, Authors