NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

Too many weaknesses led me to choose to reject this paper. Furthermore, I believe this paper requires at least one major revision before it can be considered for a top-tier conference.

1. Line 43-44. “which can lead to slow convergence and potential overfitting problems.” The authors claimed two main approaches have emerged to address the aforementioned issues. However, DoRA cannot address them [1-2]. Indeed, DoRA sometimes easily fall into not converging.
2. Line 12, “but it still necessitates a substantial number of training parameters. To address this issue…”. Line 53-86 “two significant challenges persist for these LoRA variants….To address these challenges…” In fact, I cannot understand why the authors conduct experiments in Fig.1 (a)(b). Even as the authors claimed in Line53-86, their experiment cannot answer problem 1, i.e., “the intrinsic properties of LLMs…..decomposition errors”
3. Line 248-254. It is better for the authors to provide more theoretical details on the design of activation-weight matrix and W_{aw}.
4. Line 281-290: Why NoRA’s rank allow for more complex non-linear transformations? I do not think this part “expressiveness” make sense. What can be concluded from the fact that the rank of NoRA is bounded by min(r, r′)? “Generation” part also makes no sense. And indeed, I believe the whole subsection 3.4 should be given further consideration.
5. I would like to see a comparison between LoRA and NoRA with similar parameter budgets.

Overall, the introduction of the paper is unclear, the motivation is not well-defined, the rationale behind the design in the methods section is unclear, and there are some issues with the effectiveness (Sec.3.4) of the approach.

[1] 2024, ICML, DoRA: Weight-Decomposed Low-Rank Adaptation

[2] 2024, arxiv, FLoRA: Low-Rank Core Space for N-dimension

1. Limited Scope of Comparative Analysis

• The unified design space presented in the paper is not comprehensive enough. It primarily focuses on VeRA and LoRA-XS approaches, lacking coverage of other significant approaches in this domain.
• A comprehensive table summarizing the design choices of previous works is missing. Such a table would enhance the clarity and depth of comparisons.
• The comparison provided in Figure 2 resembles an ablation study of the proposed techniques rather than a thorough comparison of prior approaches. It would benefit from including more diverse methods in the analysis.

2. Potential Compatibility with Other Approaches

• There is no discussion of the compatibility of the proposed method with orthogonal approaches such as AdaLoRA and DoRA. Exploring how NoRA could integrate with or complement these methods could provide valuable insights.

3. Theoretical and Intuitive Justifications

• The paper introduces the concept of applying a scaling matrix to mitigate decomposition errors, but the intuition behind this approach is not clearly explained. A more rigorous theoretical justification is necessary.
• For AwSVD, it remains unclear whether this technique requires a large calibration set to perform effectively.

4. Sensitivity and Practical Concerns

• The results might be sensitive to batch size, but this aspect has not been explored in detail. A discussion on how batch size affects performance would strengthen the paper.
• It is also unclear how to select input activations from the fine-tuning dataset, which could impact the practical usability of the method.

5. Rigor and Accuracy of Claims

• Several statements lack rigor and precision. For instance, the claim regarding Formula 10 suggests that a tighter rank constraint leads to more complex non-linear transformations, but this assertion is misleading. A tighter rank constraint should not necessarily imply increased complexity.
• The explanation provided in line 308 on how to approximate the parallel LoRA form by rearranging terms is vague and needs further clarification.

6. Issues with Reported Results

• Table 1 only lists DoRA and LoRA as having rank=1 which is not a practical and feasible setting.
• Additionally, the accuracy of DoRA reported in Table 2 is inconsistent with the values in the original paper. The correct accuracy should be 85.3%, not 83.0%.
• A more holistic figure illustrating performance across different ranks would provide a clearer understanding of the flexible trade-off between expressiveness and efficiency.

7. Novelty and Evaluation

• The novelty of the proposed method could be questioned, as the core idea of SVD decomposition has already been explored and analyzed by LoRA-XS and PiSSA.
• The subject-driven image generation task appears selective, possibly bordering on cherry-picking. It would be beneficial to include qualitative results on widely used benchmarks, such as DreamBooth, to ensure a more objective evaluation.

1. For instruction fine-tuning tasks, the paper only compares performance under settings with extremely low learnable parameters, which shows competitive results but falls significantly short of full-rank LoRA in performance. This raises concerns about whether the primary benefits of this work apply mainly to in-domain task transfers.
2. Is it necessary to reduce the number of optimization parameters in LoRA to save memory (particularly in the optimizer) or training time? After all, we don’t always need to compress parameter counts to such an extreme degree, and it often seems to be a trade-off. Unless it can be proven that this approach consistently outperforms various high- and low-rank LoRA fine-tuning methods, thereby serving as a superior replacement, the practical significance of this work remains in question.

• The statement regarding NoRA’s rank enabling more complex non-linear transformations lacks theoretical grounding. The discussion around “expressiveness” in that section is underdeveloped. Simply stating that NoRA’s rank is limited by \min(r, r{\prime}) does not elucidate how or why this rank impacts expressiveness. Thus, the authors should provide more experiments or theoretical explanations to demonstrate their claims.

• The decision to freeze the outer LoRA parameters to “maintain stability” lacks theoretical or empirical backing. Why this approach aids in stability needs elaboration？

• Comparative Analysis: A direct performance and parameter-efficiency comparison between LoRA and NoRA under similar parameter constraints would be insightful. This comparison would allow for a clearer understanding of how each approach performs relative to the other, given comparable resource budgets.

• In the visual section, the authors have conducted evaluations solely on a few simple classification datasets, which is insufficient. Additional complex vision-language tasks should be included, such as referring segmentation/detection, and visual caption tasks. This would better demonstrate the effectiveness of their proposed method.

1、The core of the article appears to revolve around the activation-aware matrix, which is the foundation and heart of the entire method. However, the paper seems to lack a discussion on how to confirm that the activation-aware matrix used is superior, whether there are other methods available, and how to determine whether this matrix can provide more useful information. Moreover, the approach of merely performing singular value decomposition on the activation-aware matrix and then nesting LoRA matrices might appear to offer limited innovation. 2、To better understand the contribution of each component of NoRA, such as the nested structure and AwSVD, the paper would benefit from ablation studies. These studies would isolate the effects of different design choices and provide insights into which aspects are most critical for the performance improvements observed. 3、The paper mentions that the optimal hyperparameter configurations for NoRA may vary depending on the specific task and models. This sensitivity could be a limitation for users who need to fine-tune models for different applications.