One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

We thank the reviewer for ascertaining the unified and general form of GLoRA which contributes to a novel perspective of existing PEFT methods.

Q1. The presentation of the paper could be improved, especially when comparing with previous PEFT methods. The authors could draw figures or list tables to show how existing methods can be integrated into GLoRA framework, e.g. what are the specifications of the A/B/C/D/E support tensors in Eq. (10).

A1. We thank the reviewer for the suggestion and we have worked on improving the overall presentation of the paper. Here we present a table which shows how existing methods can be integrated into GLoRA framework. We have also added this table to the main paper.. We set different support tensors to choices presented in the search space (Eq. (11)) to approximately mimic the behavior of existing methods.

Q2. I wonder how much training time (supernet training and subnet searching) does GLoRA cost, such that to compare with existing methods more clearly from the perspective of training efficiency.

A2. When compared to other adaptation methods, the benefits gained with GLoRA in terms of performance improvement as well as generalization across datasets significantly outweigh the additional time spent in the adaptation process.

We demonstrate the superiority of GLoRA by analyzing the performance improvement per unit training time of GLoRA vs. other counterparts. Quantitatively, GLoRA requires an additional 5.6 folds of training time compared to a single run of LoRA amounting to a total of ~142 minutes for each VTAB-1k task. The GPU memory consumption of GLoRA is 13 GB compared to 9 GB for LoRA. Most of it is primarily because GLoRA requires roughly 5 times more epochs than LoRA for appropriate convergence and the additional time is spent on the evolutionary search process. This extra time of GLoRA leads to an average increase of 4.5 % accuracy across 19 vision tasks as compared to LoRA. Beyond this, the biggest benefit comes in terms of generalization across datasets. Additionally, we consider 5 best-performing methods on the VTAB-1k dataset - LoRA, NOAH, FacT, SSF and RepAdapter; and pick the task-specific best-performing models across them. We denote the training time of LoRA as x and the corresponding training time of NoAH, FacT, SSF and RepAdapter are 6x, x, 1.2x and 0.9x respectively. GLoRA's total training time including architecture search is 6.6x. The table below shows the performance and training time of Best-5 methods and GLoRA. This is to indicate that the combined training time of 5 best existing methods is more than that of GLoRA and yet it delivers a superior performance over its counterpart. It is important to note that this gain is reported assuming that the other adaptation methods do not require any hyperparameter search. However, unlike GLoRA, needing minimal hyperparameter search (see Appendix C), some of the other adaptation methods require thorough data-specific hyperparameter search for optimal performance. If we consider this aspect, then the training time required by GLoRA would be significantly lower in a relative sense. We have mentioned the actual training time and memory of GLoRA in the appendix of the main paper.

Q3. There exist some typos: 1) "PETL"(maybe PEFT?) at the end of page 3 (first line of Sec. Limitations); 2) 4th line of page 4: "wieght" -> weight.

A3. We thank the reviewer for pointing out these typos and we have updated the draft accordingly.

Q4. How about the performance if we do not add the weight/bias scaling term: W_0 x A and D x b_0 in Eq. (10) ? Or else, which of the five tensors are really necessary in terms of efficiency and efficacy ?

A4. In the construction of GLoRA architectures, each support tensor plays an important role and is necessary to achieve the optimal performance. Among the various supports, A is utilized to scale the weight. B has the role to scale the input and shift the weight. C is the layer-wise prompt serving a similar function of VPT-Deep, D and E are used to scale and shift the bias, respectively. In the table below, we show the performance on CIFAR dataset by setting some support tensors to None, and it is clear that the performance dip is observed for all cases. Clearly, omitting any of these tensors gives sub-optimal performance, implying that each of these is important.

In terms of inference efficiency, GLoRA enjoys zero computational overhead due to the complete structural re-parameterization framework (as shown in Table 4). In terms of training efficiency, it is governed by the set of supports that form the GLoRA architecture. Among LoRA, Vector, Scalar and None attributes, LoRA is most computationally most expensive due to matrix multiplication involved in it, Vector being the next due to dot product operations and Scalar is the most efficient due to simple scalar multiplication. Additionally, since D and E cannot be LoRA, these are the most efficient support tensors. Further, C is the least efficient one due to the absence of scalar multiplication. We have also added these details in the updated draft.

Thank the authors for their rebuttal, which has resolved my concerns. I would like to keep my original rating.

Q3. The authors touch on memory and training time costs but fail to provide concrete figures to substantiate their claims. Offering detailed, quantitative data on these costs would provide readers with a clearer picture of GLoRA's practical ramifications.

A3. When compared to other adaptation methods, the benefits gained with GLoRA in terms of performance improvement as well as generalization across datasets significantly outweigh the additional time spent in the adaptation process.

We demonstrate the superiority of GLoRA by analyzing the performance improvement per unit training time of GLoRA vs. other counterparts. Quantitatively, GLoRA requires an additional 5.6 folds of training time compared to a single run of LoRA amounting to a total of ~142 minutes for each VTAB-1k task. The GPU memory consumption of GLoRA is 13 GB compared to 9 GB for LoRA. Most of it is primarily because GLoRA requires roughly 5 times more epochs than LoRA for appropriate convergence and the additional time is spent on the evolutionary search process. This extra time of GLoRA leads to an average increase of 4.5 % accuracy across 19 vision tasks as compared to LoRA. Beyond this, the biggest benefit comes in terms of generalization across datasets. Additionally, we consider 5 best-performing methods on the VTAB-1k dataset - LoRA, NOAH, FacT, SSF and RepAdapter; and pick the task-specific best-performing models across them. We denote the training time of LoRA as x and the corresponding training time of NoAH, FacT, SSF and RepAdapter are 6x, x, 1.2x and 0.9x respectively. GLoRA's total training time including architecture search is 6.6x. The table below shows the performance and training time of Best-5 methods and GLoRA. This is to indicate that the combined training time of 5 best existing methods is more than that of GLoRA and yet it delivers a superior performance over its counterpart. It is important to note that this gain is reported assuming that the other adaptation methods do not require any hyperparameter search. However, unlike GLoRA, needing minimal hyperparameter search (see Appendix C), some of the other adaptation methods require thorough data-specific hyperparameter search for optimal performance. If we consider this aspect, then the training time required by GLoRA would be significantly lower in a relative sense. We have mentioned the actual training time and memory of GLoRA in the appendix of the main paper.

We thank the reviewer for poinitng out GLoRA's commendable capacity to generalize across diverse tasks and the cohesive mathematical formulation to adapt to new tasks.

Q1. Structural re-parameterization requires storing the full set of weights (including bias) for every individual downstream task. This means that as the number of these tasks increases, the storage needs can become prohibitively large. Although this approach might improve inference performance, the substantial storage overhead can be a major impediment for real-world deployment, especially when multiple tuned-models are needed to handle different downstream tasks.

A1. We implement structural re-parameterization by storing a single base network and multiple GLoRA support tensor weights for each downstream task. Structural re-parameterization of GLoRA takes negligible time (<5s across 100 trials on Llama-7B) and hence depending upon the downstream task, the appropriate support tensors can be re-parameterized in the base network almost instantly during runtime. This saves the storage requirement as GLoRA support tensors require <1% storage as compared to base model and gives a great deployment/inference time saving as shown in Table 4.

Q2. The clarity of the paper is occasionally compromised by abrupt topic transitions, such as the unexpected introduction of "GLoRA with Higher Capacity" in section 2.6, without prior elucidation of terms like H_i and H_ini. A more coherent and gradual introduction of these concepts would enhance readability.

A2. We have revised our draft to make the topic transitions smoother. Further, we describe a brief elucidation of terms like H_i and H_uni. The Vapnik-Chervonenkis Dimension (VC Dimension) is a measure of the capacity (complexity, expressiveness) of a set of functions that can be learned by a statistical classification algorithm. It is defined as the cardinality of the largest set of points that the algorithm can shatter. By estimating the VC Dimension of a deep model, we can get an idea of how capable the model is of fitting complex datasets. A very high VC Dimension could indicate that the model has enough capacity to learn the training data perfectly but might overfit and generalize poorly on new data.

Definition of VC Dimension. The VC Dimension of a hypothesis class $\mathcal H$ (a set of functions) is the largest number of points that can be shattered by $\mathcal H$. A set of points is said to be shattered by $\mathcal H$ if, for every possible labeling (binary classification) of these points, there exists a hypothesis in $\mathcal H$ that perfectly classifies the points according to that labeling. Mathematically, if we have a set of points $S=\{x_1, x_2, \ldots, x_d\}$, the hypothesis class $\mathcal H$ shatters $S$ if:

$

\forall y \in\{0,1\}^d, \exists h \in \mathcal H: \forall i \in\{1,2, \ldots, d\}, h\left(x_i\right)=y_i

$

The VC Dimension, denoted as $\mathbf d_\mathrm{vc}(\mathcal H)$, is the maximum size of any set $S$ that can be shattered by $\mathcal H$. If $\mathcal H$ can shatter a set of size $d$ but cannot shatter any set of size $d+1$, then $\mathbf d_\mathrm{vc}(\mathcal H)=d$. In the context of GLoRA, $\mathcal{H_i}$ denotes the hypothesis space of a randomly sampled subnet and $\mathcal{H}_\mathrm{uni}$ denotes the hypothesis space of the complete supernet.

Q3. The authors introduce multiple PEFT modules including AdaptFormer, LoRA, etc. So could GLoRA simulate all of these modules? As far as I know, these modules are applied on different layers (LoRA in multi-head self-attention layers, while AdaptFormer in MLP layers). And which layer is GLoRA applied in practice?

A3. Indeed GLoRA is capable of simulating multiple existing PEFT modules which include SSF, VPT, LoRA and RepAdapter. For example, in equation 10, setting all support tensors but A to None and setting A as LoRA would inherently give the classic LoRA model. To mimic SSF, A, D and E can be set to vector and other all support tensors to none. Similarly, C simulates the learning mechanism of a layerwise trainable prompt similar to VPT. Note that GLoRA cannot simulate AdaptFormer since it has a non-linearity in the parallel block inhibiting structural re-parameterization which is a core aspect of GLoRA. In the table below, we show how GLoRA is able to approximately mimic the behavior of many existing works by setting support tensors to specific attributes -

For vision tasks GLoRA is applied to all linear layers (MLP as well as the attention modules). In case of NLP tasks, for a fair comparison to LoRA, we apply GLoRA only in the multi-head self-attention layers. These details are mentioned in Section 3.2 of the main paper. However, in general, GLoRA can also be applied to the MLP layers, and we expect that similar to LoRA, it will improve the performance further [2].

[1] Neural Prompt Search.

[2] QLoRA: Efficient Finetuning of Quantized LLMs

We thank the reviewer for finding our work to be more flexible than existing works owing to the structural re-parameterizable framework. We would also like to appreciate the reviewer for pointing out the thorough experiments conducted showing the generality and effectiveness of GLoRA.

Q1. It seems that GLoRA is not general, since it has an evolutionary search procedure to obtain the suitable components. The idea is similar to Neural Prompt Search. GLoRA is not a fixed design as existing modules, which might limit its practicality.

A1. We understand the concern of the reviewer, however, our use of 'general' for GLoRA does not refer to one set of adapters that can be plugged on top of a model. As we have shown, the performance of such a method varies significantly across datasets. Rather, GLoRA is general in the sense that the combination of supernet training and subnet search which form an inherent part of the method, result in well performing data-specific architectures. Although GLoRA is a search based method similar to [1], the core novelty of GLoRA lies in the unified re-parameterizable formulation as presented in Eq. 10. [1] attempts to search for VPT, LoRA and Adapter in a single framework, by defining them distinctly in the supernet whereas GLoRA doesn't define any existing adapter/module in the network explicitly but proposes a much 'general formulation' which implicitly mimics the behavior of many existing works.

Regarding the limitation in terms of practicality, GLoRA identifies the right combination of support tensors/modules/prompts for each dataset. Any optimized network obtained from GLoRA can be absorbed into the base architecture using structural re-parameterization rendering superior or the same practical throughput as existing works. Thus, we do not see any limitations of the approach compared to the other baseline methods such as LoRA, SSF, etc.

Q2. GLoRA has a large search space, which might yield huge time costs. However, the authors have not mentioned the actual training time and memory cost of GLoRA, which is very important for PEFT modules.

A2. As highlighted by the reviewer, the large search space of GLoRA indeed increases the associated training time compared to the other adaptation methods. However, it is important to note that the overall time spent on the adaptation of models using GLoRA is still substantially very small when compared to fine-tuning the full model. Further, when compared to other adaptation methods, the benefits gained with GLoRA in terms of performance improvement as well as generalization across datasets significantly outweigh the additional time spent in the adaptation process.

We demonstrate the superiority of GLoRA by analyzing the performance improvement per unit training time of GLoRA vs. other counterparts. Quantitatively, GLoRA requires an additional 5.6 folds of training time compared to a single run of LoRA amounting to a total of ~142 minutes for each VTAB-1k task. The GPU memory consumption of GLoRA is 13 GB compared to 9 GB for LoRA. Most of it is primarily because GLoRA requires roughly 5 times more epochs than LoRA for appropriate convergence and the additional time is spent on the evolutionary search process. This extra time of GLoRA leads to an average increase of 4.5 % accuracy across 19 vision tasks as compared to LoRA. Beyond this, the biggest benefit comes in terms of generalization across datasets. Additionally, we consider 5 best-performing methods on the VTAB-1k dataset - LoRA, NOAH, FacT, SSF and RepAdapter; and pick the task-specific best-performing models across them. We denote the training time of LoRA as x and the corresponding training time of NoAH, FacT, SSF and RepAdapter are 6x, x, 1.2x and 0.9x respectively. GLoRA's total training time including architecture search is 6.6x. The table below shows the performance and training time of Best-5 methods and GLoRA. This is to indicate that the combined training time of 5 best existing methods is more than that of GLoRA and yet it delivers a superior performance over its counterpart. It is important to note that this gain is reported assuming that the other adaptation methods do not require any hyperparameter search. However, unlike GLoRA, needing minimal hyperparameter search (see Appendix C), some of the other adaptation methods require thorough data-specific hyperparameter search for optimal performance. If we consider this aspect, then the training time required by GLoRA would be significantly lower in a relative sense. We have mentioned the actual training time and memory of GLoRA in the appendix of the main paper.

Thank you for your kind response and encouraging comments. To provide further context, our approach only searches adaptor components but will not search for the optimal set of hyperparameters, such as rank, dropout, or alpha as in LoRA, enabling an automatic adaptation. This indicates that there is further potential to improve the performance of our framework with better configurations. We would like to address any further concerns that you might have and continue to solidify this paper.

We thank the reviewer for finding our work to consistently improve performance over previous method in both vision and language tasks as well as finding GLoRA to be more flexible, have higher capacity and provide an unified framework for PEFT.

Q1. The authors claim that GLoRA can be "seamlessly integrate into the base network", but it seems such design is for linear layer only. But there are many other type of operators like conv / normalization layers. How can GLoRA be combined with those layers?

A1. We would like to point out to the reviewer that GLoRA is an enhanced version of LoRA and the simplicity of our approach allows us to use it in a similar fashion as LoRA. We further refer the reviewer to the PEFT Library [1] which implements LoRA for CNN and other layers. Since GLoRA uses a search space as defined in Eq. 11 which comprises LoRA, Vector, Constant and None attributes; the exact PEFT implementation can be used to implement GLoRA as well. In the context of a CNN layer featuring i input channels, o output channels, and k kernels, and for the support tensor A, we establish convolutional layer weights denoted as A$_d$ (o $\times$ r $\times$ 1 $\times$ 1) and A$_u$ (r $\times$ i $\times$ k $\times$ k). In instances of LoRA with a specified rank r$_1$ < r, we employ A$_{dr1}$ (o $\times$ r$_1$ $\times$ 1 $\times$ 1) and A$_{ur1}$ (r$_1$ $\times$ i $\times$ k $\times$ k), derived through indexing from A$_d$ and A$_u$, respectively, for consecutive low-rank convolution operations. For a Vector attribute, A (1 $\times$ o $\times$ 1 $\times$ 1) is indexed from A$_d$, facilitating vector multiplication. Similarly, in the case of a constant attribute A (1 $\times$ 1 $\times$ 1 $\times$ 1), indexing is applied for scalar multiplication with the original weight matrix from A$_d$ during computation. This approach aligns seamlessly with established conventions in linear layer implementation.

Further, since normalization layers have relatively very less number of parameters, PEFT is commonly not employed for them. For example - BatchNorm2D with c channels, has only 2 $\times$ c (weight and bias) total number of parameters which is negligible as compared to Conv2D which has o $\times$ i $\times$ k $\times$ k (weight) + o (bias). Thus, unlike the conv layers, these do not require parameter-efficient adaptation.

Q2. The evolutional search (Sec 2.4) is crucial for GLoRA as it decides which layer and scheme to use during fine-tuning. However, the details of the search and final chosen paradigms are not clearly discussed in the main paper.

A2. We agree that evolutionary search is important and the corresponding details require discussion. We have described the details of the approach in Appendix B of the paper. In general, we did not observe any uniform structure of the final network across datasets. An average distribution of the network across all the datasets is presented in Figures 3 and 4 of the paper. Upon the recommendation of the reviewer we have moved details of the evolutionary search in the main paper.