AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping

We sincerely appreciate your insightful comments and valuable suggestions on our work! Below are our point-by-point responses to the issues you have raised.

W1: The authors only explore smaller ranks for LoRA and AuroRA (a maximum of 16), and while this does demonstrate AuroRA's efficacy in highly parameter-efficient scenarios it doesn't paint a full picture as to how AuroRA compares at larger ranks; for example it is fairly common for fine-tunes of larger models to use ranks as high as 64 or even 256 for more complex tasks, and the paper lacks the experimental evidence for AuroRA's performance in such use-cases.

Thank you for your very insightful question! To address your concerns, we have conducted additional experiments on more challenging mathematical tasks. Using LLaMA2-7B as the base model, we compare the performance of LoRA and AuroRA, setting the rank for both methods to 64 and the hyperparameter alpha to 128. As shown in Table H, we observe that AuroRA continues to exhibit superior performance even with a higher rank and on more complex tasks. Due to our limited computational resources, we are unable to perform tests on larger-scale models and leave this as an avenue for future work.

Table H. Performance comparison on the GSM8K and MATH datasets using LLaMA2-7B with a rank of 64.

Q1: Did the authors perform alpha tuning to determine the best LoRA, MoSLoRA and AuroRA hyperparameters for the comparison ablations, and if so what values were chosen?

Thank you for your feedback! We would like to clarify that, in accordance with prior works [1,2,3], the hyperparameter alpha is typically set to be equal to the rank or twice the rank. In our experiments, the alpha values for both LoRA and MoSLoRA strictly follow the settings from their respective original papers, which is twice the rank. For example, in the sensitivity analysis on rank, we set the alpha values for LoRA, MoSLoRA, and AuroRA to {4, 8, 16, 32} corresponding to ranks of {2, 4, 8, 16}.

Q2: Were static or dynamic weights utilised when collecting the inference results in the main body of the paper?

Thank you for your detailed comment! We wish to responsibly point out that all experimental results in the main body of the paper are collected using a "dynamic training, static merging" strategy. As observed in Table 7 (Appendix B.1), when a "fully dynamic" strategy is employed, the AuroRA-D variant actually achieves better performance compared to AuroRA, albeit at the cost of additional runtime. Furthermore, we provide a complete and detailed algorithm for AuroRA in Appendix B.2 to more clearly illustrate the processes during the training and inference stages. Finally, respectfully following your suggestion, we will be sure to emphasize this point more explicitly in the main body of our paper in a future revision.

Q3: Can the authors justify how it is mathematically possible to linearize AuroRA considering its data-dependent activations and what the implications are considering it re-introduces a linear low-rank bottleneck.

We would first like to clarify the rationale for our "dynamic training, static inference" strategy is to maximize model expressiveness during training while allowing for weight merging to eliminate any additional computational overhead at inference time. To address your concerns, we provide the following mathematical analysis of this strategy's reasonableness.

The dynamic and static forms are defined as $h_D = \mathbf{B}\sigma(\mathbf{A}x)$ and $h_S = (\mathbf{B}\sigma(\mathbf{A}))x$, respectively. We aim to demonstrate that once the model converges through dynamic training, the static form used during inference closely approximates the dynamic form, such that $h_D \approx h_S$. Let's consider the Taylor series expansion of our Adaptive Nonlinear Layer (ANL), $\sigma(\cdot)$, around the origin. Let $z = \mathbf{A}x$, we have $\sigma(z) \approx \sigma(0) + \sigma'(0)z + \mathcal{O}(\|z\|^2)$, where $\sigma(0)$ is the function's value and $\sigma'(0)$ is its Jacobian matrix at the origin. As σ is applied element-wise, its Jacobian is a diagonal matrix. For the dynamic form, substituting the expansion gives: $\mathbf{B}\sigma(\mathbf{A}x) \approx \mathbf{B}(\sigma(0) + \sigma'(0)\mathbf{A}x) = \mathbf{B}\sigma(0) + \mathbf{B}\sigma'(0)\mathbf{A}x$. In this case, the $\mathbf{B}\sigma(0)$ term functions as a constant bias, which is independent of the input $x$. For the static form, we analyze the application of $\sigma$ to matrix $\mathbf{A}$. Each element $a_{ij}$ is transformed as $\sigma(a_{ij}) \approx \sigma(0) + \sigma'(0)a_{ij}$. For the entire matrix, this yields $\sigma(\mathbf{A}) \approx \sigma(0) \cdot J + \sigma'(0)\mathbf{A}$, where $J$ is a matrix of ones. The static form then becomes: $(\mathbf{B}\sigma(\mathbf{A}))x \approx \mathbf{B}(\sigma(0)J + \sigma'(0)\mathbf{A})x = \mathbf{B}\sigma(0)Jx + \mathbf{B}\sigma'(0)\mathbf{A}x$. Upon comparison, we observe that the first-order term, $\mathbf{B}\sigma'(0)\mathbf{A}x$, is identical in both forms. For the zeroth-order term, the fixed non-linearity we adopt has $\tanh(0) = 0$, and a negligible bias $\mathcal{L}(0) = \mathbf{w}_s \cdot \mathbf{s}(0)$ exists only in the learnable non-linearity. Overall, we can consider $\sigma(0) \approx 0$. Therefore, the distinction between the dynamic and static forms mainly resides in the higher-order terms. The dynamic form preserves the complete, input-specific non-linear information for each $\mathbf{A}x$, whereas the static form "bakes in" or averages this high-order information into the merged weights. Since all learnable components have been optimized during training, this difference is minor, a point that is also empirically supported by our results in Table 7 of Appendix B.1.

Furthermore, we present an intuitive example in Appendix C, which shows that AuroRA can fit more complex matrix structures thanks to its element-wise non-linear application. This indicates that by transforming the purely linear mapping of the original LoRA into an MLP-like structure, AuroRA can capture more information during training, thereby breaking the low-rank bottleneck.

Reference:

[1] LoRA: Low-Rank Adaptation of Large Language Models, ICLR'22

[2] Mixture-of-Subspaces in Low-Rank Adaptation, EMNLP'24

[3] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, NeurIPS'24

Thank you for your detailed and constructive feedback on our paper. We provide our point-by-point responses to your comments below.

Q1: Were the authors able to reproduce the results reported from other literature?

To address your concerns, we reproduce the results for all baselines on the Commonsense Reasoning benchmarks using LLaMA3-8B as the base model. The experiments are conducted on an NVIDIA L20 (48GB) GPU, strictly following the hyperparameter configurations from [1]. The results, detailed in Table G, show that the overall outcomes are consistent with those reported in our paper. Furthermore, it is evident that AuroRA achieves the strongest performance with the minimum number of trainable parameters.

Table G. Reproduction results for LLaMA3-8B on the Commonsense Reasoning benchmarks.

Q2: Despite the experimental (ablation) evidence, I'm left wondering from a theoretical point of view why one needs a fixed non-linearity when there is already an adaptive one. The non-linearities themselves are supposed to be universal approximators; that is why they work in regression at all.

We appreciate this highly insightful question! In line with our proposed "Coarse Fitting" and "Fine Fitting", the fixed non-linearity component serves to stabilize the optimization in its early phases. This establishes a smoother optimization environment that is easier to explore, thereby avoiding the issue of converging to a suboptimal local minimum, which can occur when using a purely learnable non-linearity in very low-rank settings (such as r=2). For a detailed analysis of the approximation error, please refer to Appendix D of our paper.

References: The citation is a little lazy; first in the way multiple citations are grouped [1,2,3,4,5,...] and secondly in the way some of the literature is after the main content as a kind of afterthought. This leads to a lot of literature being cited but not discussed well. That related literature is presented after the main experiments is somewhat unscientific. It suggests that the experiments were done before doing a thorough literature search, further implying another issue: that the content is perhaps not all that novel.

Thank you for your valuable suggestion! We would like to clarify that our experiments were not conducted before our literature survey. At the same time, we will revise our manuscript following your suggestions. Furthermore, we acknowledge that due to page limitations, we were unable to discuss some related works, and we commit to expanding discussion in a future version of the paper.

Paper Formatting Concerns: Line 194-198, the complexity of matrix multiplication of A (d_out r) and B (r d_in) should be d_out r d_in? I could be wrong. Applies to all results here.

Thank you for your comment! We would like to clarify that there is no error in lines 194-198. You are correct that the complexity of an explicit matrix multiplication of $\mathbf{B}\mathbf{A}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$ is indeed $\mathcal{O}(d_{\mathrm{out}}\,r\,d_{\mathrm{in}})$. However, this explicit computation is not performed in LoRA. Instead, the operation is carried out in two sequential steps:

1. $M = X \mathbf{A}^\top$ is computed, with a complexity of $\mathcal{O}(b\,d_{\mathrm{in}}\,r)$.
2. $\Delta \mathcal{W} = M \mathbf{B}^\top$ is computed, with a complexity of $\mathcal{O}(b\,r\,d_{\mathrm{out}})$.

Reference:

[1] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning, NAACL'25

We would like to express our sincere appreciation for your time and effort in reviewing our manuscript! Here, we address your comments on a point-by-point basis.

W1: The paper acknowledges not evaluating on larger models, which would strengthen the claim of scalability.

Thank you for your valuable comment! Due to computational constraints, we are currently unable to test on larger-scale models. We leave this as a promising direction for future work.

W2: While the paper argues that the additional complexity from ANL is negligible, more concrete runtime/memory comparisons would improve practical understanding.

We appreciate your detailed comments!

We would like to respectfully point out that we provide a detailed comparison of runtimes in Table 7 (Appendix B.1), which shows that AuroRA improves performance without a significant increase in computational time.

Furthermore, to address your concerns, we compare the GPU memory footprint of LoRA and AuroRA on the Commonsense Reasoning task, using the LLaMA3-8B as the base model on NVIDIA L20 (48GB) GPU. As shown in Table C, we observe that AuroRA introduces only a minimal additional memory overhead.

Table C. Comparison of GPU memory usage on Commonsense Reasoning tasks with LLaMA3-8B.

W3: While claiming state-of-the-art, empirical comparisons with recent nonlinear variants (e.g., LoRAN, NEAT) and other PEFT methods are missing [1,2,3,4,5,6,7]. Please remove this state-of-the-art claim if these baselines are not compared.

Thank you for this valuable suggestion!

To demonstrate the performance comparison with recent non-linear variants, we compare AuroRA against NEAT [2] and SineLoRA [8] on the GLUE and Commonsense Reasoning benchmarks, using RoBERTa-Base and LLaMA3-8B as the respective base models. As shown in Tables D and E, the results indicate that AuroRA achieves optimal performance with the fewest parameters.

Table D. Performance and parameter comparison of RoBERTa-Base on the GLUE benchmark.

Table E. Performance and parameter comparison of LLaMA3-8B on Commonsense Reasoning benchmarks.

At the same time, we wish to responsibly point out that, unlike other recent non-linear variants, AuroRA transforms the purely linear structure of vanilla LoRA into an MLP-like structure. This not only introduces non-linearity but also breaks LoRA's inherent low-rank bottleneck.

We also commit to revising our claims accordingly, citing and expanding our discussion on the other PEFT methods you mentioned [3,4,5,6,7] in a future version of our paper.

W4: Some missing related works in the PEFT literature: HFT [3], LISA [4], BAdam [5], Owlore [6], Galore [7].

We appreciate you bringing this to our attention! We will amend the related work section to include these references [3,4,5,6,7] in the final manuscript.

Q1: PEFT methods are generally more useful in fine-tuning tasks, such as instruction following, math, or coding. How is AuroRA's performance in those tasks?

Thank you for your very insightful question! Firstly, we would like to point out that in Section 3.3, we apply AuroRA to the Subject-driven Image Generation task, which is a form of multimodal instruction-following. As demonstrated in Figures 4 and 6, AuroRA exhibits significantly better adherence to prompts compared to vanilla LoRA.

Furthermore, to address your concerns, we conduct extensive experiments on both math and coding tasks. The results, presented in Table F, clearly show that AuroRA possesses robust generalization capabilities and maintains superior performance across these diverse tasks.

Table F. Performance of LLaMA2-7B on math (GSM8K, MATH) and coding (Human-Eval) tasks.

Q2: In Figure 1, it is shown that AuroRA even outperforms full-parameter fine-tuning. Is there an intuitive explanation for this phenomenon?

We believe this advantage stems from the fact that AuroRA trains only a very small number of parameters (specifically, 6.18% to 25% of those in LoRA, and 0.04% to 0.06% of those in full fine-tuning). This approach prevents the catastrophic forgetting of general knowledge acquired during the pre-training phase. Furthermore, we consider this to be a demonstration of the robust stability of the AuroRA method, as it avoids overfitting to the dataset. To further illustrate this, we provide an intuitive example in Appendix C that highlights the advantages of AuroRA's non-linearity compared to vanilla LoRA.

Reference:

[1] LoRAN: Improved low-rank adaptation by a non-linear transformation, EMNLP'24

[2] Neat: Nonlinear Parameter-efficient Adaptation of Pre-trained Models

[3] HFT: Half Fine-Tuning for Large Language Models, ACL'25

[4] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, NeurIPS'24

[5] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models, NeurIPS'24

[6] Outlier-weighed Layerwise Sampling for LLM Fine-tuning, ACL Findings'25

[7] Galore: Memory-efficient llm training by gradient low-rank projection, ICML'24

[8] Sine activated low-rank matrices for parameter efficient learning, ICLR'24

We sincerely thank you for your thoughtful and detailed review of our paper. We provide our point-by-point responses to your questions below.

Q1: In the paper, you motivate AuroRA as a means to close the performance gap between LoRA and full fine-tuning by increasing representational capacity. However, in several experimental results (e.g., Table 1 and Table 3), LoRA already performs on par or better than full fine-tuning. Could you provide empirical evidence or experimental configurations where LoRA significantly underperforms full fine-tuning, and AuroRA meaningfully narrows that gap? This would help validate the motivation more clearly.

Thank you for your insightful comment! We would like to respectfully point out that in the image classification task on StanfordCars, as shown in Table 3 (Section 3.3), our method demonstrates significant improvements.

1. When using ViT-Base as the backbone model, the accuracy of Full Fine-Tuning is 79.8, while LoRA achieves only 45.4. In contrast, our proposed AuroRA reaches an accuracy of 75.7 while using only 12.7% of LoRA's parameters, successfully narrowing this performance gap.
2. The case for ViT-Large is analogous. We observe a significant performance gap where LoRA's accuracy (73.3) is considerably lower than that of Full Fine-Tuning (88.9). Our AuroRA model substantially mitigates this gap, reaching 82.5 accuracy while using only 12.5% of LoRA's parameters.

Q2: Several recent nonlinear LoRA variants are cited (e.g., LoRAN, LoDA, NEAT, SineLoRA), but not included in the empirical comparisons. To better contextualize the performance of AuroRA, can you add a comparison against these methods on at least one representative NLP and CV benchmark? This would clarify whether your proposed nonlinear design offers substantive improvements over existing nonlinear extensions to LoRA.

To address your concern, we compare our method against two recently proposed non-linear baselines: NEAT [1] and SineLoRA [2]. We conduct these comparisons on the GLUE and Commonsense Reasoning benchmarks, using RoBERTa-Base and LLaMA3-8B as backbone models, respectively. As the results in Table A and Table B below demonstrate, AuroRA achieves the highest performance with the minimum number of parameters:

Table A. Performance and parameter comparison of RoBERTa-Base on the GLUE benchmark.

Table B. Performance and parameter comparison of LLAMA3-8B on Commonsense Reasoning benchmarks.

Q3: A key advantage of LoRA is that the learned low-rank matrices can be merged into the base weights, eliminating inference overhead. Since AuroRA includes nonlinearities (e.g., tanh, B-spline), this merging is no longer possible. Can you include an inference-time benchmarking comparing AuroRA to LoRA to demonstrate that the additional runtime cost is minimal or acceptable in practice?

Thank you for your detailed comment! We wish to clarify our methodology, as discussed in Appendix B.1, AuroRA uses a "Dynamic Training, Static Inference" strategy, allowing for weight-merging that results in zero extra overhead during inference. We empirically support this choice in Table 7 (Appendix B.1) , which shows that our strategy strikes the optimal balance between accuracy and time consumption when compared to fully dynamic (AuroRA-D), fully static (AuroRA-S), and standard LoRA approaches. For this reason, we adopt this strategy in all experiments in the main body of the paper. Furthermore, we provide a thorough analysis of the computational complexity during training in Section 2.4 and present the complete algorithmic workflow of AuroRA in Appendix B.2 for full transparency.

Reference:

[1] Neat: Nonlinear Parameter-efficient Adaptation of Pre-trained Models

[2] Sine activated low-rank matrices for parameter efficient learning, ICLR'24