From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients

We would first like to thank you for your time to review our work. We would now like to address your weakness and questions one by one as follows:

Weakness 1

The primary purpose of our gradient dynamics study is to establish and explain the non-uniform emergence of low-rank properties across different LLM weight matrices as an artifact of non-uniform gradient dynamics. Note that, ignoring the initialization of weight matrices, the resultant weight matrix at the end of pre-training is essentially a prolonged accumulation of gradients, which explains the necessity of non-uniform rank decomposition. We agree with you that we do not need pre-training information for deciding layer-wise rank in WeLore and it can be identified by singular value thresholding. Our gradient dynamics study reveals an significantly important property of LRCs that they are not only very compressible but also very good at fine-tuning on downstream tasks. In our work, WeLore’s two primary contributions of compression and memory-efficient finetuning receive its interpretability from the gradient dynamics (e.g., why some layers are low-rank?, why LRCs are more tunable?). Note that the observations we have from pre-training a small scale LLaMa-130M translate very well for pre-trained checkpoints of LLaMa-7B, and thereby make WeLore generalizable.

Weakness 2 and Question 4

We would like to clarify your weakness part-by-part and support it with some additional experiments. (1) Based on the gradient dynamics, we found that the gradients of the N-LRCs are primarily saturated and don't offer informative gradient signals from data for corresponding weight matrix update. This provide an opportunity to trim down N-LRCs during fine-tuning to avoid non-informative and noisy gradient signals, and still be able to match the fine-tuning. (2) For Figure 1 and 8, we have indeed used the WeLore style adaptive and non-uniform rank reduction technique to first create a 50% compressed model, and then perform full-finetuning and WeLore-style N-LRCs finetuning keeping exactly the same hyperparameter settings for fair comparison. (3) As recommended by you, we provide additional experiments (setting - full LLaMa-7B model with full-finetuning and WeLore style fine-tune where we backpropagate only layers where >90% rank reduction is possible with <1e-3 reconstruction error). We can clearly observe that with only ~27% of the total layers, we can closely match/outperform the performance of full-finetuning and outperform LoRA. Note that this experiment doesn’t perform any compression on the layers of LLaMa-7B but instead only selectively finetune some layers.

Weakness 3

Thank you for raising your doubt. Firstly, we agree with you that the degradation at aggressive effective rank reduction of 50% is high (7.03 -> 11.87), but we would like to highlight that it is notably lower than SoTA rank reduction techniques (e.g., SVD-LLM which achieve 12.42 and ActSVD which achieve 26.39 perplexity on C4 validation set despite using calibration dataset) by adopting a non-uniform rank reduction strategy. Secondly, regarding your doubt of comparing LoRA 1st column (8.21 PPL) with WeLore 3rd column (17.87 PPL) at ~27,000 MB memory; we would like to highlight that WeLore contribution is two-sided - a joint compression and memory efficient fine-tuning technique. We believe that your argument of comparing a 30% compressed finetuning with LoRA with 60% compressed finetuning with WeLore is not fair from the compression angle. A 30% compressed LoRA finetuned model will have higher computation tax during inference wrt. 50% compressed WeLore. While we agree that WeLore requires comparatively higher memory than LoRA during fine-tuning, it is significantly lower than full-finetuning. Our goal of comparison is centered around within a reasonable memory requirement, how well WeLore based finetuning of x% compressed model compare with LoRA.

Question 1

WeLore allows determination of non-uniform rank reduction ratio for different weight matrices within LLMs. These ratios can be used as a prior while performing activation aware low-rank decomposition methods which take into account activation outliers while weight matrix decomposition. Our experiments (Table 2) found significant performance benefits when adopting WeLore non-uniform rank ratio for this technique. While we agree that activations are indeed connected to weight matrices (which are accumulation of the gradient dynamics over the course of pretraining), we require more clarification from the reviewer about the kind of reading expected for the transferability to better respond to this question.

Dear Reviewer 9G7z:

Given that there is only few hours left before the pdf update window closes, we would deeply appreciate response from you for our rebuttal. If you feel that your concerns have been resolved, we would greatly appreciate it if you could consider raising the score. If you have any further question, we would love to address them within the limited time window.

Thank you once again for your time. We hope you have a wonderful day!

Best wishes,

The Authors

We would first like to thank you for your time to review our work. We next want to address your weakness and questions one-by-one as follows:

Weakness 1: Lacking a clear linkage between gradients and ranks.

Thank you for raising this point. Our work empirically establishes that low-rank properties across gradients are not uniform across all the components of LLMs. Through our experiments on pre-training a small scale LLaMa-130M, we showed that gradients of some components saturate early during training and don't change over time. To clarify the clear linkage between gradients and rank of a layer, we state that: Given that we ignore the initialization of weights, weights are inherently a result of cumulative addition of gradients over the course of pretraining. Layers with early saturation of gradients, didn’t receive enough diverse gradients updates to develop high-quality low rank properties during pre-training. On the other hand, layers which in turn receive diverse low-rank gradients for a prolonged period tend to converge with low-rank compression properties. We will further elucidate this in our revised draft.

Weakness 2: Comparison with SVD-LLM baseline.

Thank you for your suggestion. In the following table, we provide comparison with SVD-LLM for perplexity on the C4 dataset on LaMMa-7B. Note that, SVD-LLM is a uniform rank reduction strategy that relies on calibration data for compression, on the other hand WeLore doesn’t require any calibration data for simplicity, speed and eliminating the high sensitivity of activation-based SVD on calibration datasets.

Under fair settings, WeLore non-uniform reduction ratio can significantly benefit from incorporating the usage of calibration data as shown in the following table. This illustrate the significant usefulness of a non-uniform rank reduction ratio strategy provided by WeLore, which can benefit majority of the SoTA novel calibration-dependent and calibration-independent SVD techniques.

Weakness 3/Question 3: Proposed fine-tuning method is not a general fine-tuning method. Could the proposed fine-tuning method be applied to fine-tune uncompressed models or models compressed by other methods?

We politely disagree with your point. WeLore fine-tuning strategy asserts that, in case of limited memory resources, fine-tuning by backpropagating only though the layers whose gradients behavior haven’t saturated (i.e., LRCs layers) can be highly fruitful and closely mimic full-finetuning. We empirically illustrated that layers which are more low-rank friendly are also better tunable due to their continuously changing gradient dynamics. The proposed fine-tuning can also be used independent of compression and we have conducted additional experiments as following to validate that WeLore observations can be used in isolation for fine-tuning uncompressed models as well. In the following experiments, we illustrate that with only finetuning ~27% of layers (WeLore LRCs) we can closely match/outperform the performance of full-finetuning and outperform LoRA. Note that this experiment doesn’t perform any compression on the layers of LLaMa-7B but instead only selectively finetune some layers.

Question 1: What is the primary challenge addressed in this paper: rank determination or the relationship between gradient dynamics and rank determination?

Thank you for your question. Firstly, WeLore investigates the non-uniform emergence of low-rank properties across the weight matrices of LLMs as a consequence of varying gradient dynamics across them which accumulate over the course of pre-training within weight matrices (note that ignoring the initialization, weights are essentially the cumulation of gradients). Secondly, WeLore provides an easy-to-implement and data-agnostic strategy of using singular value based thresholding to determine the non-uniform rank reduction ratio for different layers. Thirdly, based on the gradient dynamics of low-rank friendly layers, WeLore provides an extensively experimentally validated fine-tuning strategy for fine-tuning in memory constrained settings where it suggests to back-propagate only through the LRCs layers (layers which are more compressible are also more tunable) for maximum gain. Experiments illustrate that it can closely match and sometimes surprisingly outperform full-finetuning.

Dear Reviewer uJ8A:

Given that there is only few hours left before the pdf update window closes, we would deeply appreciate response from you for our rebuttal. If you feel that your concerns have been resolved, we would greatly appreciate it if you could consider raising the score. If you have any further question, we would love to address them within the limited time window.

Thank you once again for your time. We hope you have a wonderful day!

Best wishes,

The Authors

We would first like to thank you for your time to review our work. We appreciate your comments that our paper is well-written and our analysis is novel. We would like to address your weakness and questions as follows:

Weakness 1:

Thank you for raising this point. While the submission primarily explores WeLore from the fine-training perspective, over the course of rebuttal, we have the opportunity to explore how WeLore can be useful during pre-training. WeLore experiments illustrate that gadients of some layers saturate during training quite early (we call this as early-bird saturation) and don't accumulate diverse gradients from the training data. We explored: If the gradients of some LLM components are saturated, do we actually need to further spend our computational resources on them during pretraining? To this end, we additionally conducted experiments to freeze such layers (average 0.9 cosine similarity of gradients over 100 iterations of pre-training) after their gradients saturate during the rest of pretraining. Very interestingly, we found that the final performance of LLaMa-130M doesn’t significantly varies (freezed validation PPL = 24.623 v/s no-freezing validation PPL = 24.942) which is a indication that these layers holds marginal importance after gradient saturation during rest of the pre-training duration. We will indeed provide more experimental results for extension of WeLore in the pre-training setting in our final version.

Question 1:

Thank you for your recommendation to fix the typos and we will address them.

Question 2:

Thank you for your question. We would like to highlight that Galore performs optimization across all the layers in the 30% compressed checkpoint and requires an optimizer state (although in low-rank format) for each component. On the other hand, WeLore recommends only to fine-tune/backpropagate through only a small fraction of LLM components (which are replaced with two low-rank matrices) while keeping the majority of components frozen during fine-tuing and thereby saving memory requirements. Based on your question, note that WeLore based selective layer fine-tuning strategy can also be adopted for GaLore, and we will conduct additional experiments to validate its effectiveness.

We would first like to thank you for reviewing our work and finding it enjoyable. We indeed agree with your comment about the necessity to carefully study the underlying training dynamics of LLMs and exploiting the observations/clues from them to develop better training recipes, architectures, etc. which can improve the computational and memory bottleneck of LLMs.

Over the course of rebuttal, we have the opportunity to explore how WeLore can be useful during pre-training. WeLore experiments illustrate that gadients of some layers saturate during training quite early (we call this as early-bird saturation) and don't accumulate diverse gradients from the training data. We explored: If the gradients of some LLM components are saturated, do we actually need to further spend our computational resources on them during pretraining? To this end, we additionally conducted experiments to freeze such layers (average 0.9 cosine similarity of gradients over 100 iterations of pre-training) after their gradients saturate during the rest of pretraining. Very interestingly, we found that the final performance of LLaMa-130M doesn’t significantly varies (freezed validation PPL = 24.623 v/s no-freezing validation PPL = 24.942) which is a indication that these layers holds marginal importance after gradient saturation during rest of the pre-training duration. We additionally agree with your suggestion to have some theoretical arguments on weight subspace, and this can be a default extension for our submission which we are already considering.