NOLA: Compressing LoRA using Linear Combination of Random Basis

Thanks for your valuable feedback.

Addressing Weaknesses:

Language experiments:

• Yes, full finetuning is worse which is consistent with the results in LoRA paper. We believe it is due to overfitting.

• This is an interesting suggestion. we looked at the literature and could not find any prior work reporting zero-shot results for this E2E NLG benchmark. We tried it ourselves and got very low accuracy. We believe we need to engineer a prompt for this task since a non-finetuned model does not know the task. We will do this for the camera ready version.

Meanwhile, we found the following paper [b] that shows zero-shot results on a related dataset, DART. They too report very low accuracy for the zero-shot setting: Their BLEU score is 13.26 for zero-shot and 33.41 for 3-shot. We also report the results for DART in the appendix, but our results are not directly comparable with theirs since our experiments are with GPT-M and GPT-L while they use GPT-XL.

[b] Keymanesh, Moniba, Adrian Benton, and Mark Dredze. "What makes data-to-text generation hard for pretrained language models?." arXiv preprint arXiv:2205.11505 (2022).

Vision experiments:

1. We agree and thank the reviewer for the suggestion. We perform experiments with NOLA and LoRA on various more challenging datasets. The results are reported in the following table and also in Table 6 of the revised paper. These are more challenging settings given an ImageNet pretrained model since the accuracy for some cases is around $30$% to $40$%. Interestingly, NOLA is on-par or better than LoRA in all cases with almost 3 times smaller number of parameters, so our conclusion holds on these more challenging datasets too.

5 shot:

10 shot:

2. We report the information on number of parameters of the linear layer here and in the caption of Table 5 of the revised version. The size is: 0.01M, 0.1M, 0.2M, 0.1M parameters for CIFAR-10, CIFAR-100, CUB and Caltech respectively. Similarly, the linear classifier sizes for the additional datasets considered in the revised submission are: 0.03M, 0.2M, 0.1M, 0.3M for Aircraft, Food101, Pets and SUN397 datasets respectively. We did not previously include this in the table since the linear layer classifier is constant for all methods and makes the table cluttered.

3. Continued in Next Comment...

Thanks for your valuable feedback.

Addressing Weaknesses:

1. This is a great question that we have addressed in subsection named "Why Fewer Parameters Matter?" in the Introduction of the original submission. At the time of submission, we were envisioning that there will be several finetuned LLMs that need to be stored and used for inference. Interestingly a version of this vision is already here with the introduction of "GPTs" by OpenAI a couple of weeks ago. Task-specific GPTs are very easy to generate by the users, so we expect to see a large collection of them soon in the market place. Hence, handling the storage of those models and more importantly transitioning between them at the inference time is an interesting challenge that we suggest to solve. We argue that if the number of parameters are very small (e.g., in NOLA), we can store the weights for several GPTs in the GPU memory itself reducing the latency due to transferring them from CPU memory to GPU as soon as a new query for a specific GPT arrives. Then, even a small reduction in the size of the model compared to LoRA can become important due to the high cost of GPU memory and growing number of GPTs.

Moreover, note that the sizes are smaller in these GPT-2 experiments compared to more advanced LLMs. Implementing LoRA and NoLA on larger LLMs will result in much bigger number of parameters. Hence, any reduction in size compared to LoRA will be important. Note that LoRA has a lower bound on the number of parameters corresponding to $r=1$ while NOLA is flexible and can trade accuracy with the number of parameters independent of the rank or feature dimension.

Interestingly, a very recent paper [a] published a couple of weeks ago on arXiv (following) suggests that handling many task-specific LLMs is an important challenge and introduces a computer architecture solution. We are addressing a similar challenge by reparameterization at the learning time.

[a] Sheng, Ying, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou et al. "S-LoRA: Serving Thousands of Concurrent LoRA Adapters." arXiv preprint arXiv:2311.03285 (2023).

Addressing Questions:

We did try the pseudo random generator with the same seed on several GPUs that we had access to (2080Ti, Quadro-6000, Quadro-8000, V100, P100, RTX-6000-Ada, and 3090). In all cases, the generated random sequences were the same, suggesting that these GPUs use the same random generator. Note that one can always run the random generator on the software (Python code) rather than relying on the efficient hardware implementation. Hence, even if the generator at a new device does not match with the one used in training, we can ship the parameters of the generator and run it on the software instead.

Thanks for your valuable feedback.

Addressing Weaknesses:

1. PRANC reparameterizes CNN models for visual recognition tasks while training from scratch. We use a similar reparameterization but with several key distinctions:

(a) We introduce a similar reparameterization to both A and B matrices of LoRA method and show on-par results with much smaller number of parameters.

(b) We learn the residual model instead of learning it from scratch as in PRANC.

(c) Moreover, PRANC is computationally expensive and has a large memory footprint due to the large basis matrix. We resolve this issue in NOLA since the big matrix is factorized into two smaller matrices. In the appendix, we show NOLA outperforms PRANC on vision tasks from scratch too.

(d) In addition, in Figure 2, we show that the factorization in NOLA offers much higher coverage of the subspace compared to PRANC with similar number of parameters.

2. In the Appendix of the original submission, we directly compare NOLA with PRANC on vision tasks (Table 8 for CIFAR-10 and Table 9 for ImageNet-100). NOLA outperforms PRANC with on-par or smaller number of parameters. We report this in Appendix due to space limitations. PRANC is not suitable for language tasks due to its high cost of computation and memory for large models. We present the comparison with PRANC on ImageNet-100 below. The `Trained model' in the table refers to training the entire ResNet-18 model from scratch.

Addressing Questions:

1. Yes, when using non-shared basis, the training is almost $7$% slower compared to using shared basis. In shared basis case, the same basis are used across all the layers. We believe the extra training time is due to the latency of the random matrix generator since in the non-shared case, we call it once for each layer at each iteration. Table 2 shows that shared basis version works as well as the non-shared one.

2. The goal of that ablation study and Table 3 is to show the effect of changing the rank while keeping the number of parameters constant. This is possible in NOLA since we decouple the number of parameters and the rank. We show that the results do not change significantly with varying rank suggesting that in this setting, the rank does not necessarily add to the expressiveness of the model. Note that such a study is not possible with LoRA since increasing rank increases the number of parameters too. Here and in the revised version, we also report a new section in the table that shows the results when we vary the number of parameters while keeping the rank constant. As expected, the model improves with more number of parameters.

3. $A$ and $B$ are not quantized, but we do not need to keep them in the memory. In the inference for a layer, we can calculate $A$ and $B$ using quantized $\alpha$ and $\beta$ and discard $A$ and $B$ before going to the next layer. Then, there will be very small memory overhead for storing $A$ and $B$ for a single layer. Hence, the memory will be almost the same as LoRA with similar number of parameters. Moreover, if the original model is not quantized, the memory savings with quantized LoRA during inference is minimal and may not matter.

4. Thanks for the suggestion. Note that simply highlighting the best is not enough since we are not claiming that NOLA is consistently better than LoRA in terms of accuracy. We want to show that they are on-par while NOLA is smaller in size. Hence, in the revised Table 5, we "boldface" the best number and "underline" any method within one point accuracy difference of the best method to show that NOLA and LoRA are close. Hope this helps.

5. Since PRANC matrices are initialized with random values and are not optimized, we need to combine several of them linearly (with learned coefficients) to represent $\Delta W$.

Thanks for your valuable feedback.

We argue that there are many possible local minima for $A$ and $B$ matrices in LoRA method. Training LoRA results in finding one of those local minima. Inspired by PRANC, our main intuition is that one of those local minima may live in a low dimensional random subspace. Hence, in learning, we directly look for that local minimum instead of searching for the whole space. Since the subspace is low-dimensional, we save on the number of parameters. Our experiments support our intuition empirically as we do not loose accuracy compared to original LoRA while reducing the number of parameters and decoupling them from the rank in LoRA method. We will clarify this in the text.

As stated, our main conclusion from the experiments is that our method can reduce the number of parameters while getting on-par results compared to LoRA. Since the numbers are on-par with LORA, we do not specifically highlight the best numbers on Table 1. We believe a part of confusion for the reviewer on Table 1 is due to showing that different variants (MLP, QV, etc) of our method achieves on-par results too. We will move some of those variants to the appendix or ablation section in the camera ready.

Addressing Questions:

1. Yes, they are similar on LoRA paper too. One explanation may be that GPT-2 M has enough capacity for these tasks, but we are not sure about a clear explanation. This may need to be studied as future work.

2. That's a good idea. We changed the order in the revised version to make it more clear.

3. We perform this experiment where the number of parameters for NOLA and LoRA are the same (0.35M). The results are shown below and in Table 1 of the revised paper. Our results are again on-par with LoRA suggesting that one can use NOLA instead of LoRA even if the size of LoRA is suitable for the application. Note that we never claim that our method outperforms LoRA in language tasks. We simply state that our method is capable of reducing the size while the results are on-par with LoRA.