Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape

The paper presents results on common datasets (GLUE, InstructEval, MT-Bench) for NLP tasks as well as small-scale qualitative T2I experiments to provide sufficient evidence for Flat-LoRA’s effectiveness.

The mathematical derivations appear correct to the best of my knowledge.

To prove that the proposed method improves generalization, the authors analyze it in multiple experimental setups, following the widely accepted frameworks and using the same metrics and datasets. As the improvement over most of the benchmarks is minute, it can be beneficial to provide statistical analysis to justify the difference, e.g., in Tables 1 and 2. Improvement in T2I task is shown only qualitatively and requires additional analysis using common datasets (i.e., measure CLIP-T/CLIP-I on DreamBench).

The paper references relevant prior work and discusses concurrent approaches. The proposed method complements these existing approaches and can be effectively combined with them.

Some recent PEFT methods are missing, including GaLore [1], CorDA [2].

[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection [2] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning

• Methods:

  • The paper introduces Flat-LoRA, which optimizes low-rank adaptation within a flat region of the full parameter space. This addresses the problem of standard LoRA, which may find solutions that are sharp in the full parameter space, potentially harming generalization.
  • Flat-LoRA uses a Bayesian expectation loss objective with random weight perturbations to smooth the loss landscape, promoting convergence to flatter minima. This is a computationally efficient alternative to sharpness-aware minimization (SAM).
  • The method includes a refined random perturbation generation strategy, considering weight magnitude and model width scaling, to improve generalization performance.
  • The paper also introduces a method for storing random seeds to ensure memory efficiency.

• Evaluation Criteria:

  • The paper compares Flat-LoRA with other LoRA variants, including PiSSA, LoRA-GA, DoRA, AdaLoRA and LoRA+. Flat-LoRA can be seamlessly integrated with previous approaches.
  • The paper evaluates Flat-LoRA on diverse tasks: natural language understanding, image classification, dialogue generation, mathematical reasoning, coding abilities, and text-to-image generation.
  • The models were evaluated based on performance metrics suitable for each task, such as accuracy for natural language understanding and image classification, first-turn score with GPT-4 for chat tasks, accuracy for math tasks, and PASS@1 metric for code tasks.
  • The study includes out-of-domain generalization experiments using corrupted datasets and instruction-following benchmarks.
  • Ablation studies are conducted to assess the impact of different LoRA ranks and perturbation variance.
  • The memory and time costs of Flat-LoRA are compared to those of LoRA.
  • The loss landscape is visualized to demonstrate that Flat-LoRA achieves a flatter loss landscape compared to LoRA.

• Appropriateness:

  • The methods seem appropriate for the problem. Flat-LoRA directly addresses the limitations of LoRA by seeking flatter minima in the full parameter space, which is expected to improve generalization. The use of Bayesian expectation loss and refined perturbation strategies aims to achieve this efficiently.
  • The evaluation criteria are also well-suited. The tasks cover a wide range of applications relevant to large language models and computer vision models. The use of both in-domain and out-of-domain datasets, along with ablation studies, provides a comprehensive assessment of Flat-LoRA's performance and robustness.
  • The evaluation includes comparison with SAM, showing that Flat-LoRA achieves comparable or superior performance to SAM while requiring less memory and training time.

Yes, I checked proposition 3.2 and its derivations in Equations 8 and 9. But I have not thoroughly verified Equation 9. Also, Not sure how it behaves with the dimensionality of W.

• Hyperparameter Tuning and Selection: The paper mentions specific values for the random perturbation strength $\sigma$ for different models and datasets. While an ablation study on $\sigma$ is presented in Appendix C, it's focused only on CIFAR-10/100 with CLIP ViT-B/32. The paper lacks a thorough justification for the chosen values of $\sigma$ for other tasks and models. It's possible that the reported improvements are sensitive to this hyperparameter, and the selected values might not be optimal across all settings. A more systematic approach to hyperparameter tuning, or at least a wider exploration of $\sigma$ values across different experiments, would strengthen the validity of the results.

• Comparison with Baselines: While Flat-LoRA is compared to standard LoRA and other LoRA variants, the depth of the comparison could be enhanced.

  • For instance, in the comparison with other LoRA variants (Table 5), the paper shows consistent improvements. However, it doesn't delve into whether these improvements are statistically significant. Authors have Reported confidence intervals but no explanation was provided around them in Section 4.6.
  • The comparison with SAM (Table 6) highlights the memory and time efficiency of Flat-LoRA. However, the choice of perturbation radius $\rho$ for SAM is based on a limited exploration ($\rho \in \{0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.5\}$). It's possible that a different value of $\rho$ could lead to better performance for SAM, potentially altering the conclusions of the comparison.
  • The paper mentions stronger baselines achieved for Llama-2-7B compared to previous work. While this is a positive aspect, it also raises the question of whether the relative improvements of Flat-LoRA would hold against even stronger, more recent baselines that might have emerged since the submission of this work.

• Out-of-Domain Generalization Analysis: The out-of-domain experiments on corrupted CIFAR-100-C and instruction following are valuable. However, the analysis could be more nuanced.

  • For CIFAR-100-C (Figure 4), the performance gains increase with corruption severity. While this suggests Flat-LoRA's robustness, a statistical analysis of these gains would be beneficial to confirm their significance at different corruption levels.
  • For instruction following (Table 4), the improvements on DROP and Human-Eval are highlighted as more pronounced. A discussion on why Flat-LoRA shows a greater advantage on these specific tasks compared to MMLU and BBH could provide more insight into the method's strengths.

• Loss Landscape Visualization: The loss landscape visualizations (Figure 6) are qualitative. While they visually suggest a flatter landscape for Flat-LoRA, it's important to acknowledge that these are projections along random directions in the high-dimensional parameter space. The flatness observed in these 2D projections might not fully capture the characteristics of the loss landscape in all relevant directions. A more quantitative measure of flatness, if feasible, could complement these visualizations.

• Scope of Applicability: While the paper demonstrates Flat-LoRA's effectiveness across various tasks, the underlying reasons for its varying degrees of improvement across different modalities and datasets are not thoroughly investigated. Understanding the conditions under which Flat-LoRA provides the most significant benefits would be valuable for future research and application.

In conclusion, while the experimental design is broad and covers multiple aspects, a more critical perspective reveals potential limitations in the thoroughness of hyperparameter tuning, the statistical rigor of baseline comparisons, the depth of out-of-domain analysis, and the quantitative nature of the loss landscape evaluation. Addressing these points could further strengthen the claims made in the paper.

The key contributions of the Flat-LoRA paper are significantly related to the broader scientific literature in the field of parameter-efficient fine-tuning and the understanding of loss landscapes in deep learning. Specifically, Flat-LoRA builds upon and extends prior work in Low-Rank Adaptation (LoRA) and methods for finding flat minima. Here's a breakdown of how Flat-LoRA connects to the literature:

• Building upon Low-Rank Adaptation (LoRA): Flat-LoRA directly addresses a potential limitation identified within the LoRA framework [Hu et al., 2022]. While LoRA efficiently reduces the number of trainable parameters by optimizing low-rank matrices, the authors of Flat-LoRA observed that a flat minimum in the LoRA optimization space might still correspond to a sharp region in the full parameter space. This idea is illustrated in Figure 1 and further supported by Figure 6 and the discussion in Section 3.2. Flat-LoRA aims to improve upon standard LoRA by explicitly considering the flatness of the loss landscape in the full parameter space, which is a novel perspective compared to many existing LoRA enhancements.

• Addressing Limitations of Sharpness-Aware Minimization (SAM) in the Context of LoRA: The paper discusses Sharpness-Aware Minimization (SAM) [Foret et al., 2020], a well-established technique for improving generalization by seeking flat minima. The authors acknowledge the potential of integrating SAM with LoRA (LoRA-SAM) [Li et al., 2024a]. However, they highlight several limitations: SAM applied to LoRA parameters only optimizes sharpness in a restricted space, SAM requires an additional gradient step, doubling training cost, and computing sharpness in the full parameter space with SAM is memory-intensive. Flat-LoRA offers an alternative approach to achieving flat minima that aims to overcome these computational and memory overheads.

• Leveraging Bayesian Expectation Loss and Random Weight Perturbation (RWP): Flat-LoRA's core idea of using a Bayesian expectation loss objective [Duchi et al., 2012; Bisla et al., 2022] to smooth the loss landscape and pursue flat minima is rooted in prior work. This line of research suggests that minimizing the expected loss under random weight perturbations can lead to better generalization. Flat-LoRA builds upon this by designing a refined random perturbation generation strategy that considers the filter structure and input dimension, differentiating it from simpler RWP methods like Gaussian Model Perturbation (GMP) [Wang & Mao, 2021] or basic random noise injection [Wu et al., 2022; Li et al., 2024b]. The memory efficiency of Flat-LoRA, achieved by storing only the random seed and filter norms, directly addresses a key concern when applying perturbation-based methods to parameter-efficient fine-tuning.

• Connection to the Broader Understanding of Flat Minima and Generalization: The paper's motivation stems from the widely held belief that flat minima in the loss landscape are linked to improved generalization and robustness. Flat-LoRA contributes to this understanding by proposing and demonstrating a method to find flatter minima in the full parameter space specifically within the context of efficient fine-tuning using LoRA. The experimental results, particularly the improved out-of-domain generalization on corrupted datasets and instruction following, provide empirical support for this connection in the context of Flat-LoRA.

• Orthogonal to Other LoRA Enhancement Techniques: The authors explicitly state that Flat-LoRA's approach of optimizing the sharpness of the loss landscape in the full parameter space is orthogonal to other proposed methods for enhancing LoRA performance, such as adaptive rank allocation (AdaLoRA [Zhang et al., 2023a, 2023b]), decomposition of weight updates (DoRA [Liu et al., 2024]), improved initialization (PiSSA [Meng et al., 2024]; LoRA-GA [Wang et al., 2024]), and learning rate adjustments (LoRA+ [Hayou et al., 2024]). The experiments in Section 4.6 demonstrate that Flat-LoRA can be seamlessly integrated with some of these methods (e.g., Flat-PiSSA, Flat-DoRA), leading to further performance improvements, highlighting its potential as a complementary technique.

In summary, Flat-LoRA contributes to the scientific literature by:

• Identifying a limitation of standard LoRA regarding the sharpness of the loss landscape in the full parameter space.
• Proposing an efficient alternative to SAM for finding flat minima in the context of LoRA, leveraging Bayesian expectation loss and a refined random perturbation strategy.
• Demonstrating improved generalization (both in-domain and out-of-domain) across a wide range of tasks and model sizes.
• Offering a memory and computationally efficient method that can be readily integrated with existing LoRA training pipelines and is orthogonal to other LoRA enhancement techniques.

NA

The authors conducted experiments on datasets spanning multiple domains, including Natural Language Understanding, Image Classification, Large Language Model, and Text-to-Image Generation. These benchmarks are well-matched to the research scope, making them appropriate for evaluating generalization.

n/a – The paper does not include formal theoretical proofs beyond conceptual justifications.

I have reviewed all the experiments. The experimental design is well-structured and covers various domains and tasks. However, an additional ablation study on perturbation strength ($\sigma$) in larger models would help assess the generalizability of the approach.

The paper provides a simple yet effective method for improving LoRA training by leveraging perturbation-based optimization to encourage flatter minima.

The paper adequately discusses relevant related works.

The paper evaluates performance on standard benchmarks (GLUE, text/image tasks, code generation, etc.), making comparisons against strong LoRA baselines and other variants. This breadth of experiments aligns well with the parameter-efficient fine-tuning literature.

The authors argue that smoothing the loss in the full weight space leads to broader basins and better generalization. While the Bayesian expectation loss formulation is standard, the paper’s main theoretical link is that random perturbation akin to filter-wise noise effectively regularizes the final solution.

Experiments are structured clearly, with appropriate baselines and repeated trials. The paper carefully examines both standard benchmarks and out-of-domain shifts (e.g., CIFAR-100-C corruption, instruction-following tasks). One limitation: results compare final performance but do not show training curves or highlight differences in train versus test loss. Explicit train–test curves or a direct measure of the generalization gap would strengthen the evidence that improved performance primarily stems from better generalization.

Flat-LoRA extends the line of work on PEFT techniques by bringing insights from random perturbation based regularization and sharpness-aware minimization to improve generalization. There is not much that is particular to LoRA or PEFT beyond the observation that it can also be prone to sharp minima.

No crucial references appear missing.