GeoLoRA: Geometric integration for parameter efficient fine-tuning

We thank the reviewer for their constructive feedback. We are pleased that the strengths highlighted by the reviewer align with the aspects we prioritized when writing our article, and we are glad these points came through clearly. We hope our responses to the weaknesses will convince you to raise your score and accept our work.

1. GeoLoRA validation:

We recognize the importance of larger-scale evaluations; however, given our computational resources and the fact that we average our results over multiple runs, we believe the chosen benchmarks are already considerably large. We also wish to point out that a main drawback of current LoRA-type adapters is their sensitive choice of hyperparameters, which GeoLoRA does not exhibit. We have illustrated this behavior in the Ablations Section on page 9 as well as in Figure 2. Here, using hyperparameters where AdaLoRA fails to reach satisfactory accuracy, GeoLoRA can still reach good results. This is a clear example of a setting where AdaLoRA fails, whereas GeoLoRA shows good performance.

2. Baselines for GLUE:

Please note that these benchmarks are multi-objective. For clarity, we provide the average test score across all GLUE benchmarks for each method, where it becomes apparent that GeoLoRA has the highest average score, even when the competitors have a much higher parameter count. See Table 1 in the right columns. Beyond accuracy, other important metrics include parameter count and runtime per training and evaluation step. For instance, in the QQP benchmark, GeoLoRA does not achieve the highest accuracy; AdaLoRA achieves an improved accuracy (90.65 vs. 90.53). However, AdaLoRA requires nearly twice as many parameters (1.87 times more) and is slower in both training and evaluation. Thus, despite its accuracy advantage, AdaLoRA falls short of GeoLoRA in all other metrics for the QQP benchmark. As you can see from our new numerical experiments with fine-tuned rank budgets for AdaLoRA, its accuracy for QQP is smaller than that of GeoLoRA when a similar parameter count is chosen.

For the chosen baselines, we aimed to include the most widely used low-rank adapters currently available. Notably, AdaLoRA is among the most frequently used rank adaptive adapters at present. In addition, we have now added results for DoRA and LoRA+, since these approaches have openly available source code. The results further highlight the advantages of GeoLoRA, and we thank the reviewer for pushing us to make an effort to compare against these further baselines.

Thanks for your response. I still have the following concerns:

About Experiments

1. Larger-scale evaluations: I still believe that evaluating LoRA variants on larger datasets and models is crucial, since LoRA was originally proposed to address the substantial resource demands involved in full fine-tuning of LLMs. Based on my experience, training LLAMA-7B on a 10k subset of MetaMathQA takes approximately 7 hours using a single NVIDIA 4090. Given that there is still over a week left before the rebuttal period ends, I encourage the authors to provide additional experiments.

2. Hyperparameter Selection: Figure 2(a) and 2(b) demonstrate that GeoLoRA can converge to the intrinsic rank with different initial ranks, doesn't this indicate that GeoLoRA is insensitive to the initial rank? The authors did not adjust the initial rank or $\tau$ in Table 6, so why did they adjust the initial rank in Table 7? Additionally, the question of $\tau$ selection remains unaddressed.

   This issue is not about ablation study; it's about the practicality of GeoLoRA (as a meticulous hyperparameter search would increase computational costs) and the fairness of comparisons. Moreover, when comparing Table 7 with Table 6, I noticed that Table 6 did not modify the initial rank or $\tau$, but it used a different learning rate, which is not mentioned in section B.1.2, along with a different number of epochs (potentially due to early stopping based on the validation set?).

   I hope the authors could clarify the questions about (1) why they chose different initial ranks even GeoLoRA is not sensitive to the initial rank (2) how GeoLoRA behaves with different $\tau$ (3) why they chose a learning rate not mentioned for other methods, does this imply the results of GeoLoRA in Table 2 are obtained with an unfair hyperparameter selection?

About Theory

Apologies for not expressing my confusion clearly in the previous round. I must admit that I am not very familiar with the theoretical aspects of the paper. The low confidence is mainly due this (the low score is mainly due to the experiments).

I agree that Section 3 provides a clear motivation and there is a solid link between Section 3 and Eq. (6). Personally, I did not find the theoretical motivations for the augmentation and truncation operations. Are these operations designed intuitively for dynamic rank allocation, or is there a deeper theoretical motivation behind them?

Minor Question

Behavior of low-rank adaptation strategies: I agree that the behavior of these strategies can be more complex in practical experiments. If the authors have the original experimental records, providing the training loss curves would be an effective way to demonstrate that GeoLoRA converges more efficiently than other methods.

Thank you for taking the time to carefully review our work and for your positive feedback. We have addressed the questions and comments you raised.

1. Clarity of Section 3:

We agree that we have dedicated a lot of space to Section 3. We felt this discussion was essential, as many researchers are unaware of this behavior, which is currently missing from the literature on low-rank adapters. To improve the structure of our article, we have added a sentence explaining the intention of Section 3 in our introduction, where we now highlight the structure of our article and point out that the main method is proposed in Section 4.

2. Question about rank determination:

Thank you for this interesting question. We believe this is an interesting idea that could be investigated in future research. Further research on this question could also explore the use randomized SVDs and QR decompositions.

We compare with full fine-tuning (Full FT), Houlsby adapter (HAdapter), Pfeiffer adapter (PAdapter), LoRA, DoRA, LoRA+, AdaLoRA, and BitFit. We report target metrics and computational performance (higher is better) for the median of 5 runs using different random seeds. Best results per dataset are shown in bold.

We have tuned the global rank budget of AdaLoRa to match the parameter count of GeoLoRA and report the parameter counts of these two methods individually for each GLUE test case.

Thank you for carefully reviewing our work and for your feedback. We have addressed the questions and comments you raised, and we hope the revisions, clarifications, and the new results we have included will address your concerns and convince you to accept our work.

1. Performance of GeoLoRA:

The test case in Section~3 was designed to highlight potential issues with standard training through a simple numerical example that is easily understandable for readers. While we agree that the results are less pronounced for more complex benchmarks, we would like to emphasize that GLUE is a multi-objective benchmark, and we believe GeoLoRA performs exceptionally well. For clarity, we provide the average test score across all GLUE benchmarks for each method, where it becomes apparent that GeoLoRA has the highest average score, even when the competitors have a much higher parameter count. See Table 1 in the right columns.

In addition to accuracy, we emphasize that GeoLoRA shows remarkably superior performance against other key metrics, including the number of parameters and the runtime per training step and evaluation. For instance, in the QQP benchmark, GeoLoRA does not achieve the highest accuracy; AdaLoRA achieves an improved accuracy (90.65 vs. 90.53). However, AdaLoRA requires nearly twice as many parameters (1.87 times more) and is slower in both training and evaluation.

Thus, despite its accuracy advantage, AdaLoRA falls short of GeoLoRA in all other metrics for the QQP benchmark. We have added AdaLoRA runs, where we fine-tuned the rank budget so that AdaLoRA exhibits a parameter count similar to GeoLoRA. This again underlines the improved performance of our method. For example, AdaLoRA's accuracy for QQP becomes smaller than GeoLoRA's when a similar parameter count is chosen. In addition, we would like to remark that it is often observed that standard LoRA-type adapters are very sensitive to hyperparameters, which is not the case for GeoLoRA, see our ablations study on page 9 as well as Figure 2.

2. Experiment Settings:

Thank you for requesting clarification. Since reviewer XLNG raised the same question, we believe adding an explanation of why GeoLoRA has a different number of parameters for each architecture would strengthen our manuscript. This aspect is important, as it highlights a key advantage of GeoLoRA. We train all benchmarks using the same compression hyperparameter, meaning that it is not fine-tuned for each individual GLUE benchmark. While most architectures use a fixed rank, resulting in a constant number of parameters across benchmarks, AdaLoRA is rank-adaptive. However, the way the rank budget is allocated in AdaLoRA also leads to a constant parameter count for each benchmark. In contrast, a main distinction of GeoLoRA is a higher degree of adaptation and flexibility, highlighted by its ability to automatically select a different number of parameters for each benchmark, all while using the same compression hyperparameter.

For completeness, we manually tuned the rank budget of AdaLoRA to approximately match the parameter count of GeoLoRA to provide a direct comparison of the methods (see AdaLoRA, matched parameters in Table 1). We see that the proposed GeoLoRA achieves higher test scores than AdaLoRA for similar parameter counts.

Further, we have added LoRA in the Vision transformer benchmarks; see Table 2. We set the rank $r=3$ for Cifar10 and Cifar100 and $r=6$ for Tiny-Imagenet to obtain a comparable number of trainable parameters. The training hyperparameters are the same as for AdaLoRA. As expected, LoRA is outperformed by both AdaLoRA and GeoLoRA, since the parameter budget is fixed per layer and cannot be used as effectively as in the adaptive methods.

Lastly, we have added AdaLoRA to the stable diffusion benchmark, see Table 3, with scores between the ones reported for LoRA and GeoLoRa. We emphasize that both the best validation loss and the smallest parameter count with acceptable validation loss are achieved by the proposed GeoLoRA method.

3. More Baselines:

As you mentioned, there are many LoRA-type methods, and in our manuscript, we tried to identify the best-performing methods from the current literature. We focused our comparisons on AdaLoRA, since this is the most popular rank-adaptive method. Since reporting more baselines was also a request of reviewer Ht32, we decided to add Lora+ and Dora comparisons since these methods have published code available. The results further highlight the advantages of GeoLoRA, and we thank the reviewer for pushing us to make an effort to compare against these further baselines. We refer to Table 1 for the corresponding results.

Note: The Tables will be posted in subsequent comments.

We sincerely thank the reviewer for their thoughtful and detailed evaluation of our manuscript. We have carefully addressed your questions in our revisions and hope the updates clarify these aspects and further highlight the strengths of our approach.

1. Discussion of existing methods:

Thank you for pointing this out. The standard way to enforce geometry when optimizing over the low-rank manifold is to project the gradient flow on the tangent plane and then perform a discrete descent step followed by a retraction. This is essentially what the classical Riemannian gradient descent does. DLRT, the method we refer to in Table 1, is a rank-adaptive and stochastic version of Riemannian gradient descent that was specifically developed for training large networks with low-rank weights. However, a well-known problem of this classical approach is that the projection and retraction steps require a serial update of the matrix factors $U$, $S$, $V$, requiring three sequential gradient computations. With GeoLoRA we overcome this major computational bottleneck while maintaining the same strong theoretical guarantees as classical approaches. If you believe this additional context would improve the quality of the presentation, we are happy to add it to the revised paper. Please let us know if you need a more detailed explanation, which we will be happy to provide.

2. Parameters in Table 2:

You are absolutely correct. The hyperparameter controlling the number of parameters remains unchanged across different datasets, both for the baseline methods and for our proposed approach. One key advantage of our method, however, is that with a fixed hyperparameter, it automatically selects a different number of parameters depending on the dataset. As a result, you will observe varying parameter counts for GeoLoRA, even though the truncation tolerance $\tau$ is consistently set to $\tau = 0.15$. For completeness, we manually tuned the rank budget of AdaLoRA to approximately match the parameter count of GeoLoRA to provide a direct comparison of the methods. We see, that the proposed GeoLoRA achieves higher test scores than AdaLoRA for similar parameter counts. We display the extended Table below: