VeRA: Vector-based Random Matrix Adaptation

Thank you for your constructive feedback and detailed review. We have taken your suggestions and expanded our experiments to encompass not just NLP tasks but also applications in computer vision, demonstrating the versatility of our approach. Our responses below address each of your points, with parts on the vision experiments and the theoretical analysis moved to the common reply.

Adding experiments on different domains such as diffusions on CV would help.

Please see our response in the "Experiments on vision models" part of the common reply.

Theoretical analysis on the ELoRA algorithm or its limitations

Please see our response in the "Theoretical framework/expressivity of ELoRA" part of the common reply.

In Table 2, on the RTE task LoRA is significantly better than other algorithms. Is this a mistake?

Results reported for LoRA are taken from the paper that introduced the method [1]. Our reproduced results are close to this value, which may indicate that it's not a mistake.

Why are the PEFT algorithms achieving higher accuracy compared FT on the RTE dataset? Is this just overfitting?

Indeed, overfitting may be the reason of lower performance of FT compared to PEFT methods. It may also explain higher performance of ELoRA on certain tasks, as it comes with highly reduced number of trainable parameters, potentially acting as regularisation.

References

• [1] LoRA: Low-Rank Adaptation of Large Language Models, Edward J. Hu et al., 2021

Thank you for your feedback and time to review our paper. Based on this, we have further conducted more experiments on generative E2E task and MT-Bench. Please find our detailed responses to each of your points raised below:

Performance can be hurted using this method, even the evaluation is only on a limited set of GLUE benchmarks.

The primary objective of our method is to significantly reduce the number of trainable parameters, and therefore storage requirements, with comparable performance to other methods. As detailed in Figure 3, our analysis acknowledges the parameter-performance trade-off inherent in both LoRA and ELoRA methods. While in theory, ELoRA could perform worse than LoRA, in practice we see that with a sufficient number of trainable parameters, ELoRA's performance aligns with that of LoRA, despite our method using much less parameters.

In addition, we have also increased the types of evaluations by comparing our method on E2E generation (see answer below) and on instruction following tasks - additionally to the results reported in the initial version of our paper, we are introducing evaluation with MT-Bench, a well-established instruction-following benchmark. Similarly to the initial experiment, we evaluate model finetuned on the alpaca-cleaned dataset, this time taking the full training set (40k+ samples), instead of 10k subset. Results are shown in Table 1.

Table 1 Instruction tuning of more models and evaluated on MT-Bench

From this table, we make the following observations:

• Base models do not perform well compared to instruction tuned versions (scores of 2.6 vs 5.3)
• ELoRA generally closely matches LoRA's performance. In particular for the larger model sizes, such as Llama 13B the scores are very similar (5.22 vs 5.31) and for Llama2 13B ELoRA even outperforms LoRA (5.93 vs 5.77).

Consider evaluating the method on Generative tasks?

We have conducted additional experiments on a generative E2E task with GPT2, medium and large variants, finetuned with LoRA and ELoRA methods, adhering to the setup described in the original LoRA paper [1]. For LoRA we use hyperparameters reported in [1], and for ELoRA we select a rank equal to 1024 and tune the learning rate. Results after 5 epochs are reported in the following table:

As we can see in the table, our method performs on-par or better on E2E-NLG, despite using only 22%-28% of the number of trainable parameters of LoRA. We thank the reviewer for suggesting this additional experiment and will add these results to the paper.

Figure 3 and Figure 4 are each on a single but different task.

While Figures 3 and 4 are independent and address different aspects, we agree that consistency could enhance the clarity of our presentation. Consequently, we will update Figure 4 to showcase data from the RTE task, making it consistent with Figure 3.

Why not report a mean score on GLUE?

In terms of reporting on the GLUE score, our methodology is in line with that of the LoRA paper [1], which opts to report median values. This approach enables a straightforward comparison between our results and those reported in the LoRA paper and is more robust to outliers.

Why is ELoRA better than LoRA in Table 3?

One possible explanation is that ELoRA is less prone to overfitting and may achieve better results on particular tasks, especially in low-data regime. This is similar to the existing findings of PEFT methods often achieving better results than full finetuning on some tasks.

Theory about ELoRA

Please see our response in the "Theoretical framework/expressivity of ELoRA" part of the common reply.

References

• [1] LoRA: Low-Rank Adaptation of Large Language Models, Edward J. Hu et al., 2021

Thank you for your positive feedback and your suggestions to further improve the paper! Most importantly, we have now conducted experiments using Vision Transformers and shown benefits of ELoRA in terms of GPU memory. Please see below for our responses to your questions.

Have you tried extending this method to finetuning large image models?

Please see our response in the "Experiments on vision models" part of the common reply.

What are the training time benefits of your method? Is there a GPU memory benefit? Is there a latency benefit? Does your method train faster than LoRA because it has less parameters?

First, regarding inference latency, both LoRA and ELoRA exhibit no differences. In both methods, the trainable matrices and vectors can be merged into the original weights, ensuring that the inference latency remains on par with the base model. There would, however, be benefits for our lighter-weight adapters, e.g. when serving many of them concurrently [1].

To evaluate the training time and GPU memory benefits of our method, we conducted a comparison between LoRA and ELoRA while fine-tuning LLaMA 7B with the same rank (64). The results are summarized in the table below:

While ELoRA includes more operations than LoRA because of the additional vector multiplies in the forward pass, we find that it only results in a modest 1.8% increase in training time. For the GPU memory, we observe a 7.4% reduction in VRAM usage with ELoRA, as it does not require storing optimizer states and gradients for shared random matrices. We will add this to the paper and again thank the reviewer for raising this point.

References

• [1] Sheng et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. Nov 2023, ArXiv

Thank you for your feedback and your pointers and questions. Please find our answers and our newly conducted experimental results below.

It would be great to mention LiDB (Lightweight DreamBooth) in the related work section.

Thank you for pointing this out. We acknowledge the paper's relevance and will include it in our updated related work section.

Instruction tuning evaluated on a very small dataset.

While using GPT-4 as-a-judge is well-established (e.g. [1,2]), we have now run additional experiments: We have evaluated our instruction-tuned models under the well-known multi-turn evaluation, MT-Bench [1], which offers a more robust evaluation compared to the previous method and is one of the key metrics for comparing instruction-tuned methods. Similarly to the initial experiment, we evaluate model finetuned on the alpaca-cleaned dataset, this time taking the full training set (40k+ samples), instead of 10k subset. As shown in Table 1 below, we see that our model closely matches the performance of LoRA-based finetuning here too, despite using much less parameters.

Instruction tuning: Models beyond LLaMA-2 7b.

We have further increased the model diversity: Our additional experiments include two variants of LLaMA-1 and LLaMA-2, specifically the 7B and 13B models. This demonstrates ELoRA's applicability across different model sizes.

In addition, we now compare against the baseline model: We've included results for the base LLaMA 13B model without instruction tuning as a baseline [3].

Table 1 Instruction tuning of more models and evaluated on MT-Bench

From this table, we make the following observations:

• Base models do not perform well compared to instruction tuned versions (scores of 2.6 vs 5.3)
• ELoRA generally closely matches LoRA's performance. In particular for the larger model sizes, such as Llama 13B the scores are very similar (5.22 vs 5.31) and for Llama-v2 13B ELoRA even outperforms LoRA (5.93 vs 5.77).

Thank you for suggesting this. We will add these results to the paper.