GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection

In Line 50, the authors mentioned that the errors in Equation 6 are lower than those in Equation 4, as depicted in Figure 2b. This is because Equation 6 treats all attention heads collectively, thereby not accurately capturing the structure of each individual attention head. Does a lower error indicate a better scenario? Is the error calculated for each head individually or across all heads?

Response We appreciate your thoughtful suggestions. During the rebuttal period, we found the mistake of Figure 2b. We previously mistakenly calculated separate projection matrices for each attention head and used these matrices to compute the approximation error for each head. Thus, the approximation error is lower than that of exact SVD. We have addressed this mistake and revised the manuscript. With the corrected codes, the approximation error of different heads are listed as follows.

Table R4-5. Approximation Errors of Attention Heads Using Different Projection Methods.

Please note that we use the low-rank projection matrix of head 5 in the proposed method. Furthermore, GaLore $+$ reduces this approximation error by leveraging the sparsely coded residual. Additionally, due to the cross-head low-rank projection, GaLore $+$ reduces the computational complexity of GaLore's SVD from $O(h^3d_k^3)$ to $O(hd_k^2\times\log(r))$, where $d_k$ is the dimensionality of each attention head, and $h$ is the number of attention heads.

Moreover, we have checked our codes for the main experimental results and there were no mistakes.

How does GaLore+ determine the head used for computing low-ranking gradient projection? Based on my understanding, it is selected randomly. Is there any theoretical or empirical rationale behind this choice? If not, at the very least, conducting an ablation study on various head selection strategies is essential.

Response: We randomly select the head for low-rank gradient projection. We further included an ablation study on selecting different heads for the low-rank projection matrix.

Table R4-6. Approximation Errors of Attention Heads Using Projection Matrices from Different Heads.

Table R4-6 shows that choosing different attention heads does not make a significant difference.

Tebble in Line 401.

Response: Thanks for your kind remainder. We have revised the typo.

Thank you for your valuable feedbacks. Please find our answers and our newly conducted experimental results below.

The empirical results on the natural language generation tasks show insignificant improvements of GaLore $+$ over LoRa.

Response: To further demonstrate the advantages of GaLore $+$, we conducted additional experiments involving a broader range of models and lower rank settings.

Table R4-1. Fine-tuning LLaMA2-7B and LLaMA2-13B on E2E with diverse ranks.

Table R4-2. Fine-tuning LLaMA2-7B and LLaMA2-13B on CNN/DM with rank=32.

Table R4-3. Fine-tuning LLaMA2-7B and LLaMA2-13B on XSum with rank=32.

Experimental results show that GaLore $+$ consistently outperforms other methods under the same rank settings, especially when rank=32. Remarkably, in some cases, GaLore $+$ with rank=32 even surpasses the performance of LoRA with rank=128.

Thank you for your valuable feedbacks. Based on this, we have further conducted more experiments. The detailed responses to the weaknesses and questions are listed as follows.

Response to Weaknesses

[W1] Weak novelty and contribution. The techniques newly incorporated into GaLore $+$, which include cross-head low-rank projection, randomized SVD, and sparse residuals, are all essentially from existing work, which significantly weakens the overall novelty of this work. The techniques are also discussed and demonstrated solely under the scope of applying gradient projections (i.e. vanilla GaLore) and thus the contributions are too shallow, and does not seem sufficient to constitute a conference paper at ICLR.

Response: To the best of our knowledge, no prior work has explored cross-head low-rank projection, which is the main contribution of this paper. This approach offers a low-rank projection with minimal error while demonstrating significantly faster calculation compared to existing methods. Our work substantially accelerates vanilla GaLore, and we believe GaLore $+$ offers greater practical usability than its predecessor. Moreover, we believe the proposed methods can also enhance other low-rank projection techniques. We hope that our approach inspires further exploration and refinement of existing low-rank projection methods by the broader research community.

[W2] Lacking experimentation. Even though the paper is mostly empirical, the experimental setup is narrow, covering two arithmetic reasoning datasets and one natural language generation dataset. Considering that GaLore $+$ can be used essentially anywhere when finetuning pretrained Transformers, it would be great to add (1) more datasets and pretrained models as well as (2) more PEFT baselines (other than LoRA and GaLore) to fully argue that the gain in time-efficiency as well as downstream performance leads to competitive results.

Response: To further demonstrate the effectiveness of GaLore $+$, we conducted additional experiments with a broader selection of models, datasets, experimental setups, and baselines. The results are presented as follows.

Table R3-1. Fine-tuning LLaMA2-7B and LLaMA2-13B on GSM8k with diverse ranks.

[Q4] Based on Figure 3, it seems the sparsely coded residual does not have a significant impact on downstream performance. Could we compare LoRA and GaLore vs. GaLore $+$ without sparsely coded residuals to better assess the performance gains?

Response: Figure 3 in the paper demonstrates that a higher proportion of residuals generally leads to better model performance. Additionally, we compared the effects of different methods, further validating the efficacy of GaLore $+$, the results on the GSM8k and MAWPS datasets are reported in terms of accuracy, while the results on the E2E dataset are expressed as the average ROUGE scores.

Table R3-10. Fine-tuning LLaMA2-7B and LLaMA3-8B on GSM8k with rank=128.

Table R3-11. Fine-tuning LLaMA2-7B and LLaMA3-8B on MAWPS with rank=128.

Table R3-12. Fine-tuning LLaMA2-7B and LLaMA3-8B on E2E with rank=128.

Experimental results indicate that GaLore $+$ without sparsely coded residuals still achieves comparable or superior performance compared to other baselines.

Typo in Line 401: Teble 1 → Table 1

Response: Thanks for your kind remainder. We have revised the typo.

Thank you for your time and constructive comments. We would like to know whether our responses have addressed your concerns. Please feel free to comment if there are any further confusions.

Thank you authors for the commitment taken into preparing the response. Some of my questions have been addressed but I have decided to maintain my score as I still have significant concerns about the experimental results.

• Variance in ablation study results (Tables R3-7,8, and 9). Based on these results, it appears almost all performances are overlapping within one standard deviation. In other words, varying the proportion parameter for sparsely coded residual does not lead to statistically significant performance gains for proportion values from 0% up to 1.8%. I am thus not convinced whether the proposed sparsely coded residual has a significant impact on downstream performance. This variance is also not shared in Figure 3 of the main draft, which can mislead readers on the effect of sparse residuals.
• Single-seed results for main experiment (Tables 1 and 2). Connecting to W3 as well as the first weakness mentioned by Reviewer YKpf, this also makes the performance gap between GaLore and GaLore+ somewhat counter-intuitive, since GaLore+ without sparsely coded residual essentially has the same approximability as vanilla GaLore. While this could have been argued empirically instead, the main quantitative results being obtained only from a single random seed fails to support a convincing argument.

In summary, I believe the paper should be much more rigorous in its experimental setup to fully justify the two contributions mentioned in Lines 113-118. I understand this may be difficult to accomplish due to limited time, and therefore I am open to any further discussions on improving the draft from authors as well as other reviewers.

Table R3-15. Fine-tuning LLaMA2-7B and LLaMA2-13B on E2E with diverse ranks.

The experimental results show that the proposed method has acceptable variances on the performance of most experiments, and shows statistically significant improvement over the baseline methods. Moreover, we emphasize that the proposed GaLore $+$ has shown improvements on various experimental settings, including five datasets, three ranks, and three base models, compared to three baseline methods, as presented in Tables 1-4 of the revised manuscript.

Response to Weaknesses

Counter-intuitive results require explanation. While the method aims to accelerate training through randomized SVD and use the sparsely coded residuals to reduce the approximation errors, it's fundamentally still an SVD computation. Theoretically, the performance upper bound should be the exact SVD computation. Although speed improvements are understandable, the significant performance gains over exact SVD computation (GaLore) are unexpected and need explanation

Response: We appreciate your thoughtful suggestions. During the rebuttal period, we found the mistake of Figure 2b. We previously mistakenly calculated separate projection matrices for each attention head and used these matrices to compute the approximation error for each head. Thus, the approximation error is lower than that of exact SVD. We have addressed this mistake and revised the manuscript. With the corrected codes, the approximation error of different heads are listed as follows.

Table R2-10. Approximation Errors of Attention Heads Using Different Projection Methods.

Please note that we use the low-rank projection matrix of head 5 in the proposed method. Furthermore, GaLore $+$ reduces this approximation error by leveraging the sparsely coded residual. Additionally, due to the cross-head low-rank projection, GaLore $+$ reduces the computational complexity of GaLore's SVD from $O(h^3d_k^3)$ to $O(hd_k^2\times\log(r))$, where $d_k$ is the dimensionality of each attention head, and $h$ is the number of attention heads.

Moreover, we have checked our codes for the main experimental results and there were no mistakes.

The experimental methodology raises several concerns. The authors only examine cases with large rank values, despite evidence showing that LoRA typically performs best with r=8~32 during Supervised Fine-Tuning (SFT). Additionally, the single-epoch training approach may result in underfitting, potentially skewing the results. The effectiveness of the Sparsely Coded Residual method also remains uncertain, as there's insufficient evidence to show whether the improvements come from this specific component or could be achieved simply through the combination of random SVD and GaLore.

Response: Please refer to our response to questions.

Another limitation of this work is the insufficient comparison with relevant baselines. Given that the paper's main contribution centers on accelerating PEFT training, it should include comparisons with other efficient methods like DoRA and FFT LoRA that achieve similar goals with reduced parameter counts and faster training speeds.

Response: Please refer to our response to questions.

References:

[R-1] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, KwangTing Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024.

Table R2-5. Fine-tuning LLaMA2-7B and LLaMA2-13B on XSum with rank=32.

As shown in Tables R2-1 to R2-5, GaLore $+$ consistently achieves comparable or superior results across various scenarios, with its advantages being particularly notable in Arithmetic Reasoning tasks. Moreover, the proposed method achieves faster training speed than DoRA.

• Extended training epochs.

  To prevent catastrophic forgetting in fine-tuned models, we followed the common practice of using a relatively low number of epochs. However, for datasets with particularly small sizes, such as MAWPS, we increased the epoch count to 3 to ensure the model does not underfit.

  Moreover, the loss curves during training indicate that the model has converged, supporting the reasonableness of our epoch settings.

  Additionally, we compared the performance of LLaMA2-7B across different epochs, as shown in Table R2-6.

Table R2-6. Fine-tuning LLaMA2-7B on E2E with ranks and different epochs.

The experimental results in Table R2-6 show that the model performs well with just an one-epoch training, and increasing the number of epochs does not necessarily lead to improved performance.

• Comprehensive ablation studies.

  To demonstrate the effectiveness of the sparsely coded residual, we conducted ablation study on the proportion of the residual. When the proportion is set 0, the sparsely coded residual is removed and the PEFT method equals to combining cross-head projection (random SVD) with GaLore. We fine-tuned LLaMA2-7B and LLaMA3-8B models on the GSM8k, MAWPS, and E2E datasets for thorough comparison. The results for GSM8k and MAWPS are reported in terms of accuracy, while those for E2E are measured using the average ROUGE score. The experimental results are presented as follows.

Table R2-7. Fine-tuning LLaMA2-7B and LLaMA3-8B on GSM8k with rank=128.

Table R2-8. Fine-tuning LLaMA2-7B and LLaMA3-8B on MAWPS with rank=128.

Table R2-9. Fine-tuning LLaMA2-7B and LLaMA3-8B on E2E with rank=128.

The experimental results above indicate that, the sparsely coded residual can improve the performance. Furthermore, the performance of the model improves progressively as the proportion of residuals increases.

Thanks for your valuable comments. The detailed responses to the weaknesses and questions and weaknesses are listed as follows.

Response to Questions

To strengthen the paper, the authors should expand their experimental evaluation to include various rank settings, extended training epochs, comprehensive ablation studies, and detailed comparisons with modern PEFT methods. This would provide a more complete and convincing demonstration of their method's advantages.

Response: We have conducted additional experiments to address the concerns on various rank settings, extended training epochs, comprehensive ablation studies, and detailed comparisons with modern PEFT methods.

• Various rank settings and comparisons with modern PEFT methods.

  We have conducted additional experiments with low-rank settings ($r=32$) to investigate the effectiveness of GaLore $+$ under low-rank conditions. We have included two additional natural language generation (NLG) datasets (i.e., CNN/DM and XSum), a foundation model (i.e., LLaMA2-13B), and a state-of-the-art PEFT method (i.e., DoRA [R-1]) as the baseline, in evaluation. The results on GSM8k, MAWPS, E2E, CNN/DM and XSum are listed in the following five tables.

Table R2-1. Fine-tuning LLaMA2-7B and LLaMA2-13B on GSM8k with diverse ranks.

Table R2-2. Fine-tuning LLaMA2-7B and LLaMA2-13B on MAWPS with diverse ranks.

Thank you for your time and constructive comments. We would like to know whether our responses have addressed your concerns. Please feel free to comment if there are any further confusions.

Thanks for your valuable comments, we have conducted additional experiments to address the concerns on evaluation datasets, evaluation metrics, and hyper-parameters. The detailed responses to the questions and weaknesses are listed as follows.

Response to Questions

1. Given LoRA's established effectiveness in low-data fine-tuning scenarios, how does the proposed method perform under similar low-rank, data-constrained conditions?

Response: We have conducted additional experiments with low-rank settings ($r=32$) to investigate the effectiveness of GaLore $+$ under low-rank conditions. We have included two additional natural language generation (NLG) datasets (i.e., CNN/DM and XSum), a foundation model (i.e., LLaMA2-13B), and a state-of-the-art PEFT method (i.e., DoRA [R-1]) as the baseline, in evaluation. The results on GSM8k, MAWPS, E2E, CNN/DM and XSum are listed in the following tables.

Table R1-1. Fine-tuning LLaMA2-7B and LLaMA2-13B on GSM8k with diverse ranks.

Table R1-2. Fine-tuning LLaMA2-7B and LLaMA2-13B on MAWPS with diverse ranks.