Parameter Efficient Fine-tuning via Explained Variance Adaptation

Dear reviewer sfAD,

Thank you for the constructive feedback, it helps us to substantially improve our manuscript. We address the raised concerns as follows.

Novelty

We acknowledge that adaptive rank allocation and data-driven initialization have been explored in concurrent works (e.g., CorDA, LoRA-GA). However, we strongly believe that novelty should not be judged solely by whether single components used in a framework already existed, but by whether a method offers a new perspective or principled formulation that provides novel insights. That said, EVA offers a theoretically grounded approach: both initialization and rank allocation arise from maximizing expected gradient signal under a parameter constraint. This goes beyond heuristic initialization design (CorDA, LoRA-GA) or soft constraints during optimization (AdaLoRA). By proposing EVA we provide insights into the design of maximizing gradient signal without relying on hyperparameters for computing the initialization as in CorDA/LoRA-GA which allows to systematically control for rank redistribution while providing strong performance with a reduced parameter count. We believe all of these constitute a meaningful and novel contribution to the field.

Theoretical Contribution

Thank you for pointing out the lack of connection between gradient signal magnitude and generalization error. There is indeed a theoretically grounded connection between initialization via EVA and the generalization error via the Neural Tangent Kernel (NTK, [1]). To briefly recap, EVA selects a rank-r subspace $V= \operatorname{span}({v_1, \ldots,v_r})$ in which to place LoRA updates such that the expected gradient magnitude is maximized: $\mathbb{E}\_{x} \left[ \left\| \nabla_\theta f(x)^\top v_i \right\|^2 \right]$ This expectation is maximized if we choose the top eigenvectors of the activation covariance for initializing the LoRA matrices, as done in EVA. That is, EVA aligns LoRA directions with the principal components of $\Sigma = \mathbb{E}\_{x} \left[ \nabla_\theta f(x) \nabla_\theta f(x)^\top \right]$.

Intriguingly, this is equivalent to the empirical NTK matrix. In NTK theory, learning proceeds along the NTK eigenbasis. Let $K \in \mathbb{R}^{n\times n}$ be the empirical NTK matrix over training points $x\_1, \ldots, x\_n$. The NTK view tells us that at infinite width the network output evolves under gradient descent as: $f\_t(x) = f\_0(x) + K(x, X) K^{-1} (y - f\_0(X)) \cdot \left(1 - e^{-\eta \Lambda t} \right)$, and that learning decomposes across the eigenfunction of the NTK. Crucially, directions with higher NTK eigenvalues $λ\_i$​ are learned faster and more robustly [2,3].

Finally, EVA’s initialization ensures that the LoRA subspace is aligned with the top-r eigenvectors of the NTK. Let $u\_i=1n\{u_i\}\_{i=1}^n$ be the eigenvectors of K, and $y \in R^n$ be the labels. Then the squared generalization error is bounded by the residual in low-eigenvalue components: $\mathcal{E}\_{\text{gen}} \lesssim \sum\_{i=r+1}^{n} \frac{(u_i^\top y)^2}{\lambda_i^2}$. Or put differently, the projection of the residual error onto non-covered NTK directions (those outside $\mathcal{V}$) determines the generalization error. As EVA initializes the LoRA matrices with the top-r eigenvectors of the NTK, the generalization error is minimized as it is covered only by directions with strictly smaller eigenvalues. Therefore, maximizing expected gradient signal through data-driven initialization aligns low-rank adaptation directions with the top eigenspaces of the NTK. Prior work [2,3] shows that learning concentrated in these eigenspaces leads to faster convergence and lower generalization error.

Thanks to the reviewer’s comment we were able to establish this intriguing connection and we hope this satisfies their concerns as these additional theoretical insights add novelty to our proposed approach.

[1] Jacot et al., Neural Tangent Kernel: Convergence and Generalization in Neural Networks, NeurIPS 2018 [2] Arora et al., On Exact Computation with an Infinitely Wide Neural Net, NeurIPS 2019 [3] Lee et al., Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent, NeurIPS 2019

Questions:

What does the paper mean by "stability"?

Thank you for noticing the lack of definition for the term “stability”. We indeed observed that EVA can lead to smoother convergence curves compared to LoRA-GA or CorDA (Figure 3, right). However this is an empirical observation without any theoretical backing. From a theoretical viewpoint the variance in the gradients will be larger than the ones encountered in standard LoRA, as EVA concentrates gradient variance in the most informative directions. This definitely affects training dynamics and sometimes we observed that especially for higher learning rates it can be that LoRA still trains, but EVA diverges. We add a discussion on this in Section 5. We admit that the usage of this term is confusing and therefore remove it from our claims.

Since equation 12 has proportionality instead of equality, doesn't it directly follow from equation 10 after the first update step when gTg is likely non-zero?

Yes, you are correct, Eq. 12 directly follows from Eq. 10 under the statistical independence assumption. The relevance of Eq. 11 is indeed not directly given in the current form. We agree that the phrasing around this equation is a bit unfortunate. We simply meant to augment the meaning of statistical independence in a formal manner. We rephrase this to avoid any confusion at this point.

If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.

Dear reviewer LBrt,

Thank you for your valuable feedback and positive assessment of our work.

Static Dataset

We agree with the reviewer that this is a current limitation of our method, as we explicitly state in our discussion section. EVA is designed to maximize performance in the very common and practical scenario where fine-tuning is performed on a known, static downstream dataset. This is the basic setting that is considered in the literature. We believe that extending EVA to non-static datasets is a fruitful direction for future work.

Impact of Rank Redistribution

We thank the reviewer for this observation, which aligns with our own findings. As noted in our results, the performance improvement from adaptive rank allocation can indeed be task-dependent, with some domains like decision-making showing limited gains. However, a key benefit of our redistribution strategy is that it consistently reduces the number of trainable parameters across all our experiments. This is because ranks are often shifted from higher-dimensional weights to more parameter-efficient lower-dimensional ones. Therefore, even when performance is on-par, EVA offers a clear advantage in parameter-efficiency, which is particularly relevant when fine-tuning large models. For this reason, we recommend using rank redistribution by default.

Choice of Rank

We thank the reviewer for raising this point. While we acknowledge the computational overhead for very high ranks, we did explore scaling behavior beyond rank 16. Specifically, in Table 10 (Appendix A), we present results for fine-tuning Llama-2-7B with ranks up to 64. The results show that EVA and EVA+DORA are consistently the best-performing methods across all tested ranks, from 8 to 64. Furthermore, our experiments suggest that simply increasing the rank does not necessarily guarantee better performance. In fact, we empirically observed that higher ranks do not always perform better. We hypothesize this may be due to overfitting when introducing too many parameters relative to the dataset size. Consequently, EVA’s strong performance, particularly in lower-rank setups, is a key advantage for practical, compute-constrained fine-tuning scenarios, which are very prevalent in practice.

If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.

Dear reviewer G3kb,

Thank you for the constructive feedback, it helps us to substantially improve our manuscript. We address the raised concerns as follows.

Computational Fairness

We would like to clarify that the initial cost of SVD is negligible compared to the fine-tuning cost, as shown in Table 3. We rehash the aspects of the computational overhead below.

• Initialization Overhead is Minimal: The wall-clock time for EVA's SVD initialization is a very small fraction of the total training time. As shown in Table 3, for a 7B parameter model, this overhead is only 0.7% of the total training time, and we demonstrate it can be reduced to as little as 0.2% by using a smaller batch size for the SVD step. This change does not impact the quality of the initialization (Figure 10). We are confident that this minor time difference does not change the final performance of the baselines.
• Faster Convergence and Evaluation Protocol: The small upfront time investment in our principled initialization is immediately compensated by significantly faster convergence. Figures 3 and 5 clearly show that EVA achieves a lower training loss much more rapidly than LoRA and other baselines. This means that at virtually any time t after the initialization phase, EVA has already reached a more optimal state. We clarified this point in the updated version of our manuscript.

GLUE Evaluation

We thank the reviewer for this valuable suggestion regarding evaluation best practices. We agree that rigorous evaluation is paramount to estimate the generalization error. However, re-running all GLUE tasks with a revised data split was infeasible within the rebuttal period.

We would like to mention that for GLUE specifically, we applied the same comprehensive hyperparameter search protocol to all methods, including our own and all baselines. As detailed in Appendix C.3 for each method on each GLUE task, we tuned key hyperparameters (e.g., learning rates, LoRA ranks) and selected the best-performing configuration for that specific method. The final results reported in Table 2 therefore compare the optimal performance of EVA against the optimal performance of each baseline.

Questions

Initialization of B matrix

Thank you for pointing this out. It is indeed possible to initialize B as well in our setup, following works such as PiSSA or LoRA-GA. We chose the setup B=0 as there exists literature that this setting has favorable theoretical properties, such as stable convergence for larger learning rates [1]. Furthermore, there is no common consensus in the literature that one should be preferred over the other or that one has favorable properties over the other. Due to the amount of extensive experiments present in our manuscript, we aim to explore this effect in future work.

Minibatch Size

We found our initialization to be robust across different minibatch sizes. Specifically, our analysis shows that minibatch sizes between 4 and 32 for the incremental SVD computation produce highly correlated components (Figure 10, right), leading to comparable downstream performance. We did not increase the minibatch size further to ensure that the memory bottleneck is determined by the main fine-tuning stage (i.e., by the LoRA rank and training batch size), not by the EVA initialization step. This setting makes EVA's overhead negligible, adding as little as 0.7% to the total training time for our default language generation setting for Llama-2-7B on common sense reasoning tasks (Table 3).

Minimum Data for Fine-tuning

The minimum data required by EVA is determined dynamically by the convergence of principal components during incremental SVD. This convergence is controlled by a cosine similarity threshold, $\tau$. A high value for $\tau$ represents a stricter threshold for considering components converged, whereas a low value is less permissive. By default, we use $\tau=0.99$. Even for this strict setting, the initialization for Llama-2-7B is highly efficient, completing in just 80 to 300 seconds (depending on the minibatch size) after processing fewer than 100 minibatches. However, we agree that an additional analysis on the performance impact of lowering $\tau$ would be a valuable addition to the paper. We commit to including this analysis in future versions of our manuscript.

Moreover, we agree that it would be interesting to study the impact of EVA when fine-tuning with different data budgets. While we are unable to provide this analysis during the time-frame of the rebuttal, we aim to investigate this in more detail in future work. We expect that the advantage of EVA can be particularly pronounced in low-data regimes, where the effect of initialization may have an even larger role for final performance. This hypothesis is supported by the loss curves presented in Figure 3, which show that EVA achieves substantially lower loss during the early stages of fine-tuning (e.g., at 1.5K updates). Consequently, EVA may prove to be particularly effective in such low-data scenarios.

Our manuscript has improved substantially by addressing and incorporating your feedback. If any questions remain, we are happy to engage in further discussion. If you find our responses and additional results useful, we would appreciate it if you would consider changing your score.

[1] Hayou et al., LoRA+: Efficient Low Rank Adaptation of Large Models, ICML 2024

Dear reviewer 6shx,

We would like to thank you for your valuable feedback.

Novelty

We acknowledge that adaptive rank allocation and data-driven initialization have been explored in concurrent works (e.g., CorDA, LoRA-GA). However, we strongly believe that novelty should not be judged solely by whether single components used in a framework already existed, but by whether a method offers a new perspective or principled formulation that provides novel insights. That said, EVA offers a theoretically grounded approach: both initialization and rank allocation arise from maximizing expected gradient signal under a parameter constraint. This goes beyond heuristic initialization design (CorDA, LoRA-GA) or soft constraints during optimization (AdaLoRA). By proposing EVA we provide insights into the design of maximizing gradient signal without relying on hyperparameters for computing the initialization as in CorDA/LoRA-GA which allows to systematically control for rank redistribution while providing strong performance with a reduced parameter count. We believe all of these constitute a meaningful and novel contribution to the field.

We would furthermore like to highlight the practical efficiency gains resulting from our approach which is quantified in Table 3. For fine-tuning a Llama-2-7B model on a single gpu, EVA’s initialization takes only 0.7% of the total training time, whereas the data-driven methods LoRA-GA and CorDA require 2.4% and 4.5%, respectively.

Additional Baselines

We agree that the PEFT landscape is rapidly evolving. Therefore, we added a wide range of initialization methods as baselines.

DoRA:

As mentioned in line 224, we already compared EVA to DoRA on common sense reasoning (Appendix B, Table 8), mathematical reasoning (Appendix B, Table 9), vision tasks (Appendix E, Table 21), and decision-making tasks (Appendix F, Table 24). Our results show that EVA is consistently competitive. Furthermore, we demonstrate that EVA is complementary to DoRA; the combination (EVA+DORA) frequently achieves the best performance across different tasks (Appendix B, Tables 8,9; Appendix F, 24), highlighting the unique and orthogonal contribution of our initialization scheme.

MiLoRA:

We agree with the reviewer that adding MiLoRA is an important baseline due to its similarity to PiSSA. We therefore created an integration in the PEFT library [1] for it and trained it across common sense reasoning as well as math tasks. We present the results together with DoRA results in tables R1 and R2 below.

Table R1: Results on Common Sense Reasoning (mean and standard deviation across three seeds). We report results for rank 16.

Table R2: Results on Math (mean and standard deviation across three seeds). We report results for rank 16.

AG-LoRA:

Thank you for pointing out this highly relevant work. Unfortunately, the code for AG-LoRA is not publicly available, and a custom re-implementation for activation-weighting and outlier-based global rank assignment may result in subtle discrepancies to the code run by the authors. To ensure a rigorous and reproducible evaluation, we prioritize baselines with available and verified code. We will add a discussion on AG-LORA in our related work section.

GLoRA:

GLoRA is orthogonal to EVA as it does not focus on initialization. It employs evolutionary strategies to explore adapter configurations, whereas the focus of our work lies in initialization of adapters and their ranks.

In summary, we have clarified the novelty of EVA by highlighting its theoretical grounding and efficiency gains over other data-driven methods. Furthermore, we have strengthened our empirical evaluation by incorporating new results for MiLoRA, demonstrating that EVA remains highly competitive in a crowded field. We believe these clarifications and additions substantially strengthen our submission, and we hope you will reconsider your assessment of our work's contribution. If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.

[1] Mangrulkar et al., PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods, GitHub 2022