Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning

Weakness 2, Initialization Strategy

For linear layers, FLoRA requires the initialization of matrices $\mathbf{A}$, $\mathbf{G}$, and $\mathbf{B}$, with one of them being initialized to zero. We focus on the core matrix $\mathbf{G}$. If $\mathbf{G}$ is initialized to zero while $\mathbf{A}$ and $\mathbf{B}$ are randomly initialized, the results are shown in Line 1 of the table. If $\mathbf{G}$ is not initialized to zero, we consider alternative initializations such as an identity matrix, normal distribution, or orthogonal matrix, with the corresponding experimental results shown in Lines 2, 3, and 4, respectively. It can be observed that different initializations of $\mathbf{G}$ have some impact on performance. However, the influence remains within an acceptable range.

We sincerely hope that our response could address your concerns.

Dear Reviewer wiu6:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1, More Theoretical Proofs and Experiments

We sincerely appreciate your suggestions on how to improve our manuscript regarding linear layer. We hope the following responses address your concerns effectively.

Theoretical Proof

Let $\\mathbf{W}^*\\in\\mathbb{R}^{n\times m}$ be the optimal weight for a specific linear layer with frozen weight $\\mathbf{W}$, and $\\Delta\\mathbf{W}^{\*}=\\mathbf{W}^{\*}-\\mathbf{W}$ is the optimal change. Generally, the low-rank representation of LoRA’s series can be represented as $\\Delta\\mathbf{W}=\\mathbf{AGB}$, where $\\mathbf{A}\\in\\mathbb{R}^{n\\times r}$, $\\mathbf{B}\\in\\mathbb{R}^{r\\times m}$, $\\mathbf{G}\\in\\mathbb{R}^{r\times r}$ and $r\\ll\{n,m\}$, and the essential difference of these methods is the matrix pattern of $\\mathbf{G}$. In FLoRA, $\\mathbf{G}$ is arbitrary, in AdaLoRA, it is diagonal, and identity in LoRA.

Ideally, we have $\\mathbf{G}=\\mathbf{A}^{\\dagger}\\Delta\\mathbf{W}^{\*}\mathbf{B}^{\\dagger}$, where $\\mathbf{A}^{\\dagger}$ and $\\mathbf{B}^{\\dagger}$ are the Moore-Penrose Pseudo-inverses of $\\mathbf{A}$ and $\\mathbf{B}$, respectively. If $\\mathbf{G}$ is an arbitrary matrix such as FLoRA, it is obvious that no constraints are imposed on $\\mathbf{A}$ and $\\mathbf{B}$, since for any choice of $\\mathbf{A}$, $\\mathbf{B}$, we can always find a $\\mathbf{G}$. Considering when $\\mathbf{G}$ is diagonal or identity, we use SVD on \\Delta\\mathbf{W}^{\*}=\\mathbf{U} \\mathbf{\\Sigma} \\mathbf{V}^\\mathsf{T} and \\mathbf{G}=\\mathbf{A}^{\\dagger}\\mathbf{U}\\mathbf{\Sigma}\\mathbf{V}^\\mathsf{T}\\mathbf{B}^{\\dagger}. We then require \\mathbf{A}^{\\dagger}=\\mathbf{D}\_1\\mathbf{U}^\\mathsf{T} and $\\mathbf{B}^{\\dagger}=\\mathbf{V}\\mathbf{D}\_2$, where $\\mathbf{D}\_1$ and $\mathbf{D}_2$ are rectangular diagonal matrices. It means $\\mathbf{G}=\\mathbf{D}_1\\mathbf{\\Sigma}\\mathbf{D}_2$. If $\\mathbf{G}$ is diagonal like AdaLoRA, we can constrain $\\mathbf{D}\_1=\\mathbf{I}\_{r\\times n}$ and leave $\\mathbf{D}_2$ unrestricted, since any $\\mathbf{D}_2$ can correspond to a $\\mathbf{G}$. However, if $\\mathbf{G}$ is identity such as LoRA, and with $\\mathbf{D}_1=\\mathbf{I}\_{r\\times n}$, $\\mathbf{D}\_2$ must be (\\mathbf{D}\_1\\mathbf{\\Sigma})^\\dagger.

Thus, to achieve the optimal change $\\Delta\\mathbf{W}^\*$, FLoRA imposes no constraints on $\\mathbf{A}$, $\\mathbf{B}$, AdaLoRA has constraints on $\\mathbf{A}$, and LoRA further restricts $\\mathbf{A}$ and $\\mathbf{B}$. However, these constraints are not applied during training, which means that LoRA and AdaLoRA are less likely to learn the optimal change compared to FLoRA. Further, even if specific matrix patterns are constrained during training, FLoRA’s greater degree of freedom and stronger expressive power in learning make it much easier to achieve better results [1,2].

Additionally, some subsequent works have also demonstrated the effectiveness of FLoRA from different perspectives [3,4].

[1] 2023 ICLR, AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

[2] 2024 KNOWL INF SYST, Model Complexity of Deep Learning: A Survey

[3] 2024, arxiv, See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

[4] 2024, EMNLP, Mixture-of-Subspaces in Low-Rank Adaptation

More Experiments

We here report the performance of FLoRA when fine-tuning LLAMA-3-8B on commonsense reasoning tasks. As seen in the following table, FLoRA still significantly outperforms LoRA when fine-tuning LLAMA-3-8B.

Dear Reviewer VXNC:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1, Limited Scope of LLM Fine-tuning Experiments

We here report the performance of FLoRA when fine-tuning LLAMA-3-8B on commonsense reasoning tasks. As seen in the following table, FLoRA still significantly outperforms LoRA when fine-tuning LLAMA-3-8B.

Weakness 2, LoRA's Applicability in Vision-Based Models

Thank you for your valuable suggestion. We here compare several CV tailored PEFT method including last-layer tuning and visual prompt-tuning [1] on the VTAB-1K benchmark, consists of 19 image classification tasks across diverse domains. The average evaluation results are presented in the table below. Clearly, compared to methods specifically designed for computer vision tasks, FLoRA continues to demonstrate significant advantages.

[1] 2022, ECCV, Visual prompt tuning

[2] 2022, TPAMI, Neural Prompt Search

[3] 2022, ICLR, LoRA: Low-rank adaptation of large language models

[4] 2022, NIPS, Scaling & shifting your features: A new baseline for efficient model tuning

[5] 2021,EACL, Adapterfusion: Non-destructive task composition for transfer learning

[6] 2022, NIPS, Adaptformer: Adapting vision transformers for scalable visual recognition

[7] 2022, ACL, Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

[8] 2023, AAAI, Fact: Factor-tuning for lightweight adaptation on vision transformer.

[9] 2021, NIPS, Compacter: Efficient low-rank hypercomplex adapter layers

Weakness 3, Missing DoRA Results on LLaVA-1.5-7B

Thank you for your valuable suggestion. We have included the results of DoRA in the main table. Additionally, we have retrained FLoRA and updated our results accordingly.

Weakness 4, Compatibility with Weight-Decomposed Formulations

We respect your perspective; however, we believe that the core problem FLoRA aims to solve is how to achieve efficient fine-tuning while preserving structural integrity. This is fundamentally unrelated to the issue of achieving weight-decomposed formulations, which is the primary focus of DoRA. These two approaches are addressing different challenges.

Moreover, even if FLoRA is not compatible with weight-decomposed formulations, this does not imply that its design is unreasonable or its performance is suboptimal. Indeed, our experimental results demonstrate that, in the vast majority of cases, FLoRA outperforms DoRA. Specifically, in CV tasks, FLoRA’s performance significantly surpasses that of DoRA, further highlighting its effectiveness and applicability.

Weakness 5, Lack of Theoretical Grounding for the Core Space

FLoRA posits that for any dimensional parameter space, whether 2D or 4D, there exists a corresponding low-rank core space. This concept is not only intuitive but also extensively validated and applied in high-order tensor decomposition theories. For instance, CP decomposition expresses high-order tensors as a summation of low-rank tensors. Moreover, tensor nuclear norm theory establishes that for high-dimensional tensors, the optimal solution to the nuclear norm corresponds to a low-rank core tensor, which effectively captures the main characteristics of the tensor. This low-rank tensor aligns with the core space hypothesized by FLoRA.

Further supporting this concept, numerous studies have provided theoretical and experimental evidence affirming the existence of core spaces. For example, foundational research [10] has theoretically proven the existence of core spaces in linear parameter spaces, and other works [11-12] have experimentally validated their presence. As for convolutional parameters, many approaches have assumed the existence of a core space within the convolutional parameter spaces [13-14]. These findings collectively strengthen the argument that a core space exists.

[10] 2023, NMI, Parameter-efficient fine-tuning of large-scale pre-trained language models

[11] 2021, ACL, Intrinsic dimensionality explains the effectiveness of language model fine-tuning

[12] 2018, ICLR, Measuring the intrinsic dimension of objective landscapes

[13] 2024, arxiv, Large Convolutional Model Tuning via Filter Subspace

[14] 2021, NIPS, Image Generation using Continuous Filter Atoms

Weakness 6, Potential Computational Overhead

Thank you for your valuable suggestion. In fact, we have conducted extensive analyses of training costs, as presented in the Table 5 in our main text. Here, we provide the training costs for fine-tuning LLAMA3-8B. We set the batch size as 1:

These results consistently demonstrate the efficiency of FLoRA in terms of training.

Weakness 7, Focus Beyond Convolutional Kernels

We agree with your perspective. Indeed, considering that the weights of existing large models are at most four-dimensional tensors (e.g., convolutional layers), there are no corresponding large models available for testing on other higher-dimensional weight matrices, even if we wished to do so. Therefore, in our paper, we focused our evaluation on large models with convolutional layers and those with linear layers.

Weakness 8, Quantifying the Impact of Structural Integrity

We appreciate your perspective on whether preserving structural integrity truly matters in weight adaptation. In fact, LoRA’s approach of adapting a high-dimensional tensor through a two-dimensional matrix essentially disrupts its structural integrity. In contrast, FLoRA maintains the structural integrity by directly constructing a low-rank core space with the same dimensionality. Therefore, by comparing the performance of LoRA and FLoRA on convolutional layers, we can directly quantify the impact of structural integrity. We can observe that FLoRA significantly outperforms LoRA. The notable performance difference between the two methods provides strong evidence that preserving structural integrity is indeed crucial for weight adaptation in computer vision tasks. Additionally, we believe this experiment serves as the most direct evidence of the impact of structural preservation on performance.

Dear Reviewer B2Z8:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1 and 2, Convolution Core

FLoRA posits that for any dimensional parameter space, whether 2D or 4D, there exists a corresponding low-rank core space. This concept is not only intuitive but also extensively validated and applied in high-order tensor decomposition theories. For instance, CP decomposition expresses high-order tensors as a summation of low-rank tensors. Moreover, tensor nuclear norm theory establishes that for high-dimensional tensors, the optimal solution to the nuclear norm corresponds to a low-rank core tensor, which effectively captures the main characteristics of the tensor. This low-rank tensor aligns with the core space hypothesized by FLoRA.

Further supporting this concept, numerous studies have provided theoretical and experimental evidence affirming the existence of core spaces. For example, foundational researches [1-3] has theoretically and empirically proven the existence of core spaces in linear parameter spaces. As for convolution parameters, many approaches have assumed the existence of a core space within the convolutional parameter spaces [4-5]. These findings collectively strengthen the argument that a core space exists.

The learned convolution core in FLoRA helps preserve the spatial locality of convolutions because it maintains the topological structure of the original convolutional space. Unlike low-rank matrix adaptation methods (e.g., LoRA) that reshape high-dimensional convolutional kernels into two-dimensional matrices, which scatter adjacent elements and disrupt spatial relationships, FLoRA operates directly in the original tensor space. It identifies a low-rank core tensor that retains the structural integrity of the convolutional kernel. This ensures that the spatial locality, a fundamental property of convolution operations where output elements depend on localized regions of the input, is effectively preserved. Consequently, FLoRA enhances the convolution layer’s ability to process spatial information without compromising the locality inherent in its parameter space.

[1] 2023, NMI, Parameter-efficient fine-tuning of large-scale pre-trained language models

[2] 2021, ACL, Intrinsic dimensionality explains the effectiveness of language model fine-tuning

[3] 2018, ICLR, Measuring the intrinsic dimension of objective landscapes

[4] 2024, arxiv, Large Convolutional Model Tuning via Filter Subspace

[5] 2021, NIPS, Image Generation using Continuous Filter Atoms

Weakness 3, Comparison with Other Methods

Thank you for your suggestion. Following [1], we compare several CV tailored PEFT method on the VTAB-1K benchmark, consists of 19 image classification tasks across diverse domains, categorized into three groups: Natural, Specialized, and Structured. The average evaluation results are presented in the table below. Clearly, compared to methods specifically designed for computer vision tasks, FLoRA continues to demonstrate significant advantages.

[6] 2022, ECCV, Visual prompt tuning

[7] 2022, TPAMI, Neural Prompt Search

[8] 2022, ICLR, LoRA: Low-rank adaptation of large language models

[9] 2022, NIPS, Scaling & shifting your features: A new baseline for efficient model tuning

[10] 2021,EACL, Adapterfusion: Non-destructive task composition for transfer learning

[11] 2022, NIPS, Adaptformer: Adapting vision transformers for scalable visual recognition

[12] 2022, ACL, Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

[13] 2023, AAAI, Fact: Factor-tuning for lightweight adaptation on vision transformer.

[14] 2021, NIPS, Compacter: Efficient low-rank hypercomplex adapter layers

Weakness 4, Standard Deviations

Thank you for your suggestions. We have added standard deviations of the results in our manuscript.

FLoRA’s learning pattern and comparison with other fine-tuning methods (Question 4, 6)

In Figure 4, we have already compared the Frobenius norm and amplification factor metrics with other methods, such as LoRA and DoRA, and presented the trends exhibited by different methods.

It can be observed that during the training process, LoRA and DoRA amplify more task-specific features than FLoRA in the early stages. As discussed earlier, these two methods impose strong constraints on matrix patterns, resulting in a distinct directional learning pattern at the beginning, which contributes to their larger initial values. However, such constraints on matrix patterns may not always be optimal for downstream tasks, as downstream datasets often possess diverse properties. Consequently, their amplification factors upon convergence are smaller than that of FLoRA.

Therefore, FLoRA ultimately amplifies more task-specific information, which further demonstrates its superior expressive capacity.

Indeed, both Reviewer VXNC and Reviewer WIU6 acknowledged our exploration of the feature amplification factor and Frobenius norm as an insightful analysis of low-rank representation. We hope that the above response could address Reviewer UDNJ’s concerns effectively.

Weakness 5, Scalability and Computational Efficiency

We have conducted extensive analyses of scalability and computational efficiency, as presented in the Table 5 in our main text. These results consistently demonstrate the efficiency of FLoRA in terms of training.

Indeed, considering that the weights of existing prevailing large foundation models are at most four-dimensional tensors (e.g., convolutional layers), there are no corresponding large models available for testing on other extremely higher-dimensional weight matrices, even if we wished to do so. Therefore, in our paper, we focused our evaluation on large models with convolutional layers and those with linear layers.

What is the relationship between feature amplification factor and Frobenius norm of $\\Delta\\mathbf{W}$? (Weakness 4, Question 3)

In our experiments, we observed that the final value of the Frobenius norm of $\\Delta \\mathbf{W}$ is positively correlated with the Feature Amplification Factor, meaning that as one increases, the other also tends to increase. This was somewhat unexpected but also intuitively reasonable: given that the Feature Amplification Factor is directly calculated using the Frobenius norm of $\\Delta \\mathbf{W}$, it is logical to assume some intrinsic connection between the two. At the time of submission, we had not fully understood why this alignment occurred. However, with further analysis, we can now offer an explanation for this phenomenon:

Firstly, the Feature Amplification Factor is defined as the ratio of the Frobenius norm of $\\Delta \\mathbf{W}$, i.e., $\\|\\Delta\\mathbf{W}\\|\_F$, to the amount of information in the pre-trained weights $\\mathbf{W}$ projected onto the task-specific directions, denoted as $\\|\\mathbf{U}^\mathsf{T}\\mathbf{W}\\mathbf{V}\\|\_F$. Since the pre-trained weight matrix $\mathbf{W}$ is frozen during training, the value of \\|\\mathbf{U}^\\mathsf{T}\\mathbf{W}\\mathbf{V}\\|\_F depends solely on $\\mathbf{U}$ and $\\mathbf{V}$, which are two orthonormal matrices derived from $\\Delta \\mathbf{W}$ and represent task-specific directions.

In [6], the authors highlight that at any step during training, $\\Delta \\mathbf{W}$ effectively captures the task-specific directions. Consequently, from the beginning to the end of training, the changes in $\\mathbf{U}$ and $\\mathbf{V}$ are minimal (as also validated in [6]), which implies that \\|\\mathbf{U}^\\mathsf{T}\\mathbf{W}\\mathbf{V}\\|\_F remains nearly constant. In other words, a larger $\\|\\Delta\\mathbf{W}\\|\_F$ does indeed correspond to a larger Feature Amplification Factor. Therefore, while we fully respect Reviewer UDNJ’s perspective that “Frobenius norm and Feature Amplification could be incidental rather than causal,” based on the above analysis, we believe that their relationship is inherent and inevitable.

Building on our earlier analysis of the relationship between amplification factors and model performance, it is evident that a larger Frobenius norm of $\\Delta\\mathbf{W}$ corresponds to better performance. This is a surprising yet logical and exciting conclusion. In the paper, we hypothesized that larger Frobenius norm values of $\\Delta\\mathbf{W}$ allow for encoding more task-specific information, thereby effectively amplifying the task-specific signals in the frozen weights. Upon further reflection, this insight aligns with the findings of [6] but from a different perspective: amplifying task-specific information leads to better task performance. Moreover, directly increasing the Frobenius norm of $\\Delta\\mathbf{W}$ emerges as a more straightforward approach to achieve this amplification.

We sincerely appreciate the opportunity provided by the Reviewer UDNJ to explore the underlying reasons behind these phenomena together. In fact, uncovering the essence behind counterintuitive observations is the true essence of research. We look forward to further discussions and analyses of these intriguing phenomena with Reviewer UDNJ.

Is this phenomenon universal? (Weakness 3, Question 3, 5)

We used one model and one dataset in our experiments to illustrate the phenomenon and showcase our findings. Unfortunately, it is nearly impossible to theoretically prove that this phenomenon is universal. However, in our broader experiments, we observed similar trends across different datasets and models. We have updated our manuscript, and add Figure 7 in Appendix. Here we fine-tune DeBERTaV3-large on SST-2 dataset, and the results align well with our aforementioned analysis.

Furthermore, to avoid randomness, we did not selectively choose $\\Delta \\mathbf{W}$ from a specific layer of the model. Instead, we computed the mean of $\\Delta \\mathbf{W}$ across all layers. We believe this averaging approach better reflects the overall trend, providing a more comprehensive and robust representation of the observed phenomenon.

Therefore, we believe this phenomenon can be consistently observed across the experiments we conducted.

Weakness 3, 4, Question 2, 3, 4, 5, 6, Feature Amplification Factor and Frobenius Norm

We have carefully reviewed and organized the feedback from Reviewer UDNJ and identified that the primary concerns center around the Feature Amplification Factor and Frobenius Norm, comprising 2 weaknesses and 5 questions out of a total of 11 comments. These concerns are directly related to the experimental analysis section, specifically in Section 5.2, Lines 415 to 431 of the main text. Below, we provide detailed responses and hope these address Reviewer UDNJ’s concerns comprehensively.

What is feature amplification factor? (Question 2)

The Feature Amplification Factor is a metric introduced by LoRA to measure how much task-specific information is amplified by $\\Delta \\mathbf{W}$. Simply put, it calculates the ratio of the task-specific information learned by the model (represented by the Frobenius norm of $\\Delta \\mathbf{W}$) to the amount of information obtained by projecting the pre-trained weights $\\mathbf{W}$ onto the task-specific directions. Mathematically, the Feature Amplification Factor is defined as \\frac{\\|\\Delta\\mathbf{W}\\|\_F}{\\| \\mathbf{U}^\\mathsf{T}\\mathbf{W}\\mathbf{V}\\|\_F}, where $\\mathbf{U}$ and $\\mathbf{V}$ are the top $r$ left and right singular vectors of $\\Delta\\mathbf{W}$. Here, $r$ is the rank configuration of PEFT methods like LoRA.

This metric provides insight into how effectively the model adapts to downstream tasks by quantifying the relative contribution of the learned task-specific updates, and it has been widely adopted in PEFT works to evaluate the amount of task-specific information learned by the model [5-7]. If the Reviewer UDNJ has further questions regarding the definition and calculation or others of the Feature Amplification Factor, we recommend referring to Section 7 of the original LoRA paper for additional details. We also warmly welcome Reviewer UDNJ to discuss these aspects with us further.

[5] 2024, arXiv, MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Fine-tuning

[6] 2024, arXiv, Unleashing the Power of Task-Specific Directions in Parameter Efficient Fine-tuning

[7] 2024, arXiv, Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learning

What is the relationship between feature amplification factor and model performance? (Weakness 3, Question 2)

Before addressing this question, we would like to clarify that we have never stated in our paper that the Feature Amplification Factor or Frobenius Norm is directly related to task performance. We have never suggested that a higher Feature Amplification Factor directly translates to better task performance. What we consistently emphasize is that a larger Feature Amplification Factor or Frobenius Norm in FLoRA signifies that it contains more task-specific information.

Indeed, although LoRA does not explicitly state a relationship between these factors and task performance, its experimental results show a clear correlation with amplification levels. Moreover, many existing works [5-7] implicitly or explicitly suggest that a larger Feature Amplification Factor correlates with improved model performance. For instance, [6-7] directly highlights that amplifying more task-specific information can enhance model performance.

Therefore, while we did not explicitly state in our paper that there is a necessary link between the two, based on our review of existing literature and as part of our discussion with Reviewer UDNJ, we believe there is evidence to suggest a correlation: that is, larger amplification factors are often associated with better model performance.

Dear Reviewer UDNJ:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1, Question 1, Comparison with LoTR

We have already compared the differences between FLoRA and LoTR in the related work. For your convenience, we have pasted the relevant content here:

There are some works like LoTR also adopt low-rank tensor adaptations, but they are unrelated to FLoRA.

1. Usage and Impact. Based on low-rank tensor adaptation, FLoRA directly models $\Delta\mathbf{W}$ for the weight matrix of each layer, maintaining its spatial structure. However, LoTR concatenates $\Delta\mathbf{W}$ across all layers and then applies low-rank tensor adaptation, assuming all layers share same weight matrices, which is a strong assumption that leads to suboptimal performance. Besides, LoTR disrupts the original parameter space structure although they use low-rank tensor adaptation.
2. Motivation and Details. FLoRA aims to model $\Delta\mathbf{W}$ while preserving the N-dimensional topological structure, meanwhile addressing the limitations of other methods focusing only on linear weights. However, LoTR aims to achieve extreme parameter efficiency, and is tailored for linear weights, limiting its applications.

Moreover, using the same decomposition form in PEFT is indeed very common. Many works like AdaLoRA, RoSA, TriLoRA, PISSA and MiLoRA use SVD, but they hold quite distinct objectives and implementations.

We also provide the average performance of FLoRA and LoTR on NLU using RoBERTa-base in the following table. It is evident that under similar parameter budgets (0.32M), FLoRA significantly outperforms LoTR.

Weakness 2, Representations Superiority of FLoRA

Let $\\mathbf{W}^\*\\in\\mathbb{R}^{n\\times m}$ be the optimal weight for a specific linear layer with frozen weight $\\mathbf{W}$, and $\\Delta\\mathbf{W}^{\*}=\\mathbf{W}^{\*}-\\mathbf{W}$ is the optimal change. Generally, the low-rank representation of LoRA’s series can be represented as $\\Delta\\mathbf{W}=\\mathbf{AGB}$, where $\\mathbf{A}\\in\\mathbb{R}^{n\\times r}$, $\mathbf{B}\in\mathbb{R}^{r\times m}$, $\mathbf{G}\in\mathbb{R}^{r\times r}$ and $r\ll\{n,m\}$, and the essential difference of these methods is the matrix pattern of $\\mathbf{G}$. In FLoRA, $\\mathbf{G}$ is arbitrary, in AdaLoRA, it is diagonal, and identity in LoRA.

Ideally, we have $\\mathbf{G}=\\mathbf{A}^{\\dagger}\\Delta\\mathbf{W}^{\*}\\mathbf{B}^{\\dagger}$, where $\\mathbf{A}^{\\dagger}$ and $\\mathbf{B}^{\\dagger}$ are the Moore-Penrose Pseudo-inverses of $\\mathbf{A}$ and $\\mathbf{B}$, respectively. If $\\mathbf{G}$ is an arbitrary matrix such as FLoRA, it is obvious that no constraints are imposed on $\\mathbf{A}$ and $\\mathbf{B}$, since for any choice of $\\mathbf{A}$, $\\mathbf{B}$, we can always find a $\\mathbf{G}$. Considering when $\\mathbf{G}$ is diagonal or identity, we use SVD on \\Delta\\mathbf{W}^{\*}=\\mathbf{U}\\mathbf{\\Sigma}\\mathbf{V}^\\mathsf{T} and \\mathbf{G}=\\mathbf{A}^{\\dagger}\\mathbf{U}\\mathbf{\\Sigma}\\mathbf{V}^\\mathsf{T}\\mathbf{B}^{\\dagger}. We then require \\mathbf{A}^{\\dagger}=\\mathbf{D}\_1\\mathbf{U}^\\mathsf{T} and $\\mathbf{B}^{\\dagger}=\\mathbf{V}\\mathbf{D}\_2$, where $\\mathbf{D}\_1$ and $\\mathbf{D}\_2$ are rectangular diagonal matrices. It means $\\mathbf{G}=\\mathbf{D}\_1\\mathbf{\\Sigma}\\mathbf{D}\_2$. If $\\mathbf{G}$ is diagonal like AdaLoRA, we can constrain $\\mathbf{D}\_1=\\mathbf{I}\_{r\\times n}$ and leave $\\mathbf{D}\_2$ unrestricted, since any $\\mathbf{D}_2$ can correspond to a $\\mathbf{G}$. However, if $\\mathbf{G}$ is identity such as LoRA, and with $\\mathbf{D}\_1=\\mathbf{I}\_{r\\times n}$, $\\mathbf{D}\_2$ must be (\\mathbf{D}\_1\\mathbf{\\Sigma})^\\dagger.

Thus, to achieve the optimal change $\\Delta\\mathbf{W}^\*$, FLoRA imposes no constraints on $\\mathbf{A}$, $\\mathbf{B}$, AdaLoRA has constraints on $\\mathbf{A}$, and LoRA further restricts $\\mathbf{A}$ and $\\mathbf{B}$. However, these constraints are not applied during training, which means that LoRA and AdaLoRA are less likely to learn the optimal change compared to FLoRA. Further, even if specific matrix patterns are constrained during training, FLoRA’s greater degree of freedom and stronger expressive power make it much easier to achieve better results [1,2].

Moreover, some works have also demonstrated the effectiveness of FLoRA [3,4].

[1] 2023 ICLR, AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

[2] 2024 KNOWL INF SYST, Model Complexity of Deep Learning: A Survey

[3] 2024, arXiv, See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

[4] 2024, EMNLP, Mixture-of-Subspaces in Low-Rank Adaptation