Dear reviewer, thank you for the insightful and constructive feedback. We'd like to address your concerns as follows.

Q1: Overlap between Principal Weights and the original largest-magnitude weights

In Appendix G.6 of our paper, we discussed the overlap between parameters selected by LIFT and parameters selected by weight magnitude. In Fig. 16, we plotted the overlap ratio between LIFT and weight magnitude on different layer types as the number of trainable parameters varies. We showed that the overlap is generally below 20%, while different layers have different overlap ratios. We note that the overlap on MLP layers are significantly lower than that on attention layers. This suggests that the low-rank approximation of the Query and Key matrices are close to the original matrix, and Query and Key modules are low-rank in nature.

The above results indicate that the Principal Weights of LIFT and original large-magnitude weights do not overlap heavily, and how to combine the two types of weights is indeed a promising direction of future work.

Q2: Missing discussions on recent sparse fine-tuning paper.

Here we discuss the paper "Scaling Sparse Fine-Tuning to Large Language Models" [1]. This paper proposed SpIEL approach to reduce the need for computing all gradients during backward propagation, which scales the gradient memory linearly with the selected weights. While our method LIFT proposed a novel way to select Principal Weights for sparse fine-tuning and store only the optimizer states of Principal Weights to significantly reduce the memory overhead.

We also compare LIFT with SpIEL on the GSM8K dataset. We use the training setting as in Fig. 3 in our paper. For both methods, we searched learning rate among {5e-5, 1e-4, 2e-4, 5e-4} and trainable parameters corresponding to LoRA rank among {16, 32, 64, 128, 256} to obtain the best results. The table below shows that LIFT significantly outperforms SpIEL with both LLaMA-2-7B and LLaMA-3.2-3B model.

What's more, combining LIFT with the way SpIEL reduces memory overhead to further pursue memory efficiency is a promising direction for future work.

Q3: Can LIFT technique be adapted to structured sparse finetuning?

To evaluate whether LIFT can work in a structured fine-tuning fashion, we perform initial experiments on structured block-sparsity, where we select a number of $4\times4$ blocks to fine-tune (LIFT_Structured). We select 5% of all parameters to train (corresponding to LoRA rank $\approx$ 128). We choose to fine-tune LLaMA-2-7B model with MATH-10K dataset and evaluate on seven arithmetic reasoning tasks. The results are as follows.

From the table above, we can see that LIFT still achieves great performance under structured sparsity. LIFT_Structured has almost the same performance as original LIFT, while outperforming Full FT. This suggests that LIFT has the potential to be adapted to structured, sparse fine-tuning to achieve further computation acceleration.

Q4: Typos

We thank the reviewer for pointing out the typo, and we will fix it in the revised version of the paper.

References

[1] Ansell et al, 2024

Dear reviewer, thank you for the insightful and constructive feedback. We'd like to address your concerns as follows. The supplementary figures/tables are in the rebuttal link here.

Q1: Layer-wise analysis on LIFT

In Appendix G.4 of our paper, we analyze the layer-wise effect of LIFT, where we compare the performance of LIFT in fine-tuning a single type of layers. Our results show that fine-tuning only MLP layers yields significantly better results than fine-tuning attention layers. We hypothesize that attention components mostly store information on token relations rather than task-specific knowledge. On the other hand, MLP layers are more adaptive to downstream tasks, and fine-tuning them is more effective.

Based on that insight, we explore the possibility of further improving the efficiency of LIFT by only fine-tuning MLP layers. The table below shows the results of LLaMA-2-7B on arithmetic datasets where we only fine-tune the MLP layers (LIFT_MLP), and only fine-tune the attention layers (LIFT_Attn). We can see that LIFT_MLP has similar performance to the full version of LIFT, while LIFT_Attn has drastically worse performance. This suggests that for LIFT, fine-tuning MLP layers is more effective than fine-tuning Attention layers.

Q2: How does the rank-level affect the effectiveness of LIFT?

In Appendix G.5 and Fig. 15 of our paper, we study the influence of rank in low-rank approximation (LRA Rank) on the performance of LIFT. We found a correlation between LRA Rank and the number of trainable parameters, that the optimal LRA Rank increases as we select more parameters to train. In practice, when LIFT has the same number of parameters as LoRA, we find that using LRA Rank similar to LoRA rank yields optimal performance. We note that LRA Rank may differ with different models, using some layer-adaptive metrics could bring further performance gain to LIFT.

Q3: More discussions on related references

The SVFT paper [1] introduces a PEFT method called SVFT, which performs SVD on pre-trained weights and fine-tunes the resulting SVD factors. In contrast, our work considers sparse fine-tuning and proposes a novel approach for selecting Principal Weights to update. Furthermore, by storing only the optimizer states associated with these Principal Weights, LIFT significantly reduces memory overhead. We will cite SVFT in the revised version of our paper.

Q4: Detailed training settings and memory costs

Training Settings

In Fig. 3 in our paper, we showed the results of LIFT on GSM8K datasets with 4 random seeds. Due to resource limitations, for other experiments we only report the results of one seed. In Table 8 in the rebuttal link, we present the results on arithemetic datasets with four random seeds (Table 2 in the paper), which further prove the robust performance of LIFT, where it outperforms other baselines under four seeds.

Number of Trainable Parameters and GPU Memory Costs

In all experiments in the paper, we compare the best results of different methods among a range of parameter sizes. Specifically, when comparing LIFT with LoRA-like methods, we search the LoRA rank in {16, 32, 64, 128, 256}, and LIFT with the same parameter counts to ensure fair comparison, and pick their best results. In practice, we find that the LIFT and LoRA-like methods typically have the best performance at rank = 128, similar to results from PiSSA paper [2].

In Fig. 6 of Appendix B, we present the memory breakdown of LIFT compared to Full FT and LoRA under optimal settings. We showed that LIFT achieves similar memory overhead as LoRA, significantly lower than Full FT. Furthermore, we explored only fine-tuning MLP layers with LIFT, which further improves memory advantages over LoRA, while maintaining test performance (details are in the Q2 response to Reviewer ceyi).

Q5: Typos

We thank the reviewer for pointing out the typo, and we will fix it in the revised version of the paper.

References

[1] Lingam et al, 2024
[2] Meng et al, 2024

Dear reviewer, thank you for the constructive feedback. We'd like to address your concerns as follows. The supplementary figures/tables are in the rebuttal link here.

Q1: Comparison with recent sparse fine-tuning methods

We compare LIFT with sparse fine-tuning methods, SIFT [1] and SpIEL [2] (since SMT is not opensourced, we compare with other two). We will cite these papers in the revised version.

First, we compare LIFT with SIFT on GLUE tasks following the SIFT paper, with the same number of trainable parameters (5% total parameters). We use the RoBERTa-large model and search LR of SIFT and LIFT in {5e-5, 7e-5, 1e-4, 2e-4} and compare their best results. The table below shows that LIFT outperforms SIFT on all GLUE tasks, while outperforming Full FT on almost all tasks.

Second, we compare LIFT with SpIEL on GSM8K (See Q2 response to Reviewer AcQs). We show that LIFT significantly outperforms SpIEL on both LLaMA-2-7B and LLaMA-3.2-3B models.

Q2: Number of trainable parameters and GPU memory cost of LIFT

2.1. Number of trainable parameters

In all experiments in the paper, we compare the best results of different methods in a range of parameter sizes. When comparing LIFT with LoRA-like methods, we search the LoRA rank in {16, 32, 64, 128, 256}, and LIFT with the same parameter counts to ensure fair comparison. We find that LIFT and LoRA-like methods typically perform best at rank = 128.

In addition, in our analytical results (e.g. Fig. 11), we compare LIFT and LoRA both with rank = 128. Some layers of LoRA have rank larger than 128 is likely due to numerical errors when computing the rank.

2.2. GPU memory cost of LIFT

We note that although the memory cost of LIFT (as in Fig. 6 in Appendix B) is slightly larger than LoRA, we can further reduce the memory cost of LIFT while preserving performance.

In Appendix G.4, we showed that fine-tuning MLP layers is more effective than fine-tuning attention layers. The table below shows the results of LLaMA-2-7B on arithmetic datasets where we only fine-tune the MLP layers (LIFT_MLP). We see that LIFT_MLP has similar performance to LIFT. Furthermore, only fine-tuning MLP layers further reduces the memory usage on gradients and optimizer states. In Fig. 17 of the rebuttal link, we see that LIFT_MLP achieves better memory efficiency than LoRA under optimal settings.

Q3: Could LIFT negatively impact the performance on OOD tasks?

We believe although model weights undergo significant changes during fine-tuning, because only a small set of parameters are changed while most weights remain unchanged (see the center of histogram in Fig. 5), the model retains its fundamental capacities that enable it to generalize to OOD settings.

In Sec. 7.1, we also analyzed the generalization performance of LIFT by training models with arithmetic datasets and evaluating on commonsense tasks (OOD). This study verifies that LIFT achieves stronger OOD performance compared to other baselines.

Q4: Metrics like effective rank

Here we compare LIFT with HiRA [3] and evaluate the effective rank of other baselines.

The table below shows the results of LIFT and HiRA on arithmetic datasets (same as Table 2 in our paper). We search HiRA rank in {16, 32, 64, 128, 256, 512} and found that HiRA performs best at rank = 512. We see that LIFT (rank = 128) outperforms HiRA with rank = 512.

In Fig. 19 of the rebuttal link, we show the effective rank of different methods. We see that the update matrix of LIFT has larger effective rank than LoRA and PiSSA, close to Full FT, while slightly lower than HiRA. We believe although effective rank is a good indicator of model capacity, it's insufficient to soly rely on it to predict performance.

Q5: Training curve of LIFT

In Fig. 18 of the rebuttal link, we show the training loss curve of all methods in Table 2 results. We see that the convergence speed of LIFT is on par with Full FT, notably faster than other PEFT methods.

References

[1] Song et al, 2024
[2] Ansell et al, 2024
[3] Huang et al, 2025

Dear reviewer, thank you for the constructive feedback. We'd like to address your concerns as follows. Please find the supplementary figures/tables in the rebuttal link here.

Q1: Higher-level analysis on LIFT

In our paper, we conduct higher-level analysis on LIFT through empirical analysis and eigenspectrum analysis.

First, we show that parameters selected by LIFT are crucial to model performance. In Sec. 4 and Fig. 2, we show that LIFT selects principal weights that are sensitive to random perturbations, which has significant impact on the model performance, thereby validating that these weights are good candidates for fine-tuning.

Second, we also provided in-depth analysis of the eigenspace of LIFT. Sec. 7.2 and 7.3 show that LIFT induces larger weight changes (Fig. 5), and significantly increases the rank (and effective rank, see the Q4 response to Reviewer ceyi) of weight updates (Fig. 11), and exhibits greater singular subspace divergence (Fig. 10), all of these demonstrates LIFT shows a greater learning capacity in fine-tuning. Additionally, spectral norm analysis suggests LIFT enhances generalization (Fig. 8, 9), where random perturbations sharply increase the spectral norm in both random matrix and LLM settings, and LIFT's lower gradient norm curves (Fig. 12) further demonstrate its effectiveness, as supported by PiSSA [1].

Q2: Evaluation on diverse domains

We conduct more experiments on QA and Code Generation datasets. These experiments further showcase the effectiveness of LIFT on more diverse domains.

For QA, we use the experimental setup of StrategyQA dataset following recent WeLore paper [2]. For PEFT methods, we consider ranks {16, 32, 64, 128, 256}; for LIFT, we use the same counts of trainable parameters. The learning rates are {1e-5, 2e-5, 5e-5, 1e-4} for Full FT and {5e-5, 1e-4, 2e-4, 5e-4} for others. We select the best-performing config for each method and report the results below. We can see that LIFT achieves notable performance gains than all other methods on both LLaMA-2-7B and LLaMA-3-8B.

For code generation, we adopt the settings from the recent SIFT paper [3], where we fine-tune LLaMA-2-7B with Alpaca dataset for one epoch, and evaluate on the Humaneval dataset. From the table below we can see that LIFT outperforms all other methods in both pass@1 and pass@10 settings.

Q3: Is the results of LIFT robust under multiple seeds?

In Fig. 3 in our paper, we show the results of LIFT on GSM8K datasets with 4 random seeds. In table 8 in the rebuttal link, we present the results on arithemetic datasets with four random seeds (same as Table 2 in the paper), which further prove that LIFT has robust performance, where it outperforms other baselines under four seeds.

Q4: Comparison with dynamic/structured pruning methods

In Fig. 3, we have compared LIFT with a wide range of other dynamic sparse fine-tuning methods, including Top-k Magnitude, Top-k Gradient, and Movement, and showed that LIFT outperforms all other sparse fine-tuning methods. We note that unstructured sparse fine-tuning usually serves as the performance ceiling of structured and block-wise sparsity. In addition, in our main results, we have compared LIFT with S2FT, which is the state-of-the-art structured fine-tuning method by far. These results suggest that LIFT is indeed superior to other common sparse selection methods.

Q5: Hyperparameter Tuning

Here we explain the hyperparameter tuning for experiments. We will incorporate detailed information in the revised version of our paper.

Training Hyperparameters

In all experiments in the paper, we compare the best results of different methods among a range of parameter sizes. Specifically, when comparing LIFT with LoRA-like methods, we search the LoRA rank in {16, 32, 64, 128, 256}, and LIFT with the same parameter counts to ensure fair comparison, and pick their best results. We find that LIFT and LoRA-like methods typically have the best performance at rank = 128, similar to results from PiSSA paper [1].

LIFT Hyperparameters

The main hyperparameter of LIFT is the LRA Rank, which controls the rank for low-rank approximation of weight matrices. In Appendix G.5 and Fig. 15 of our paper, we study the influence of LRA Rank on the performance of LIFT. We found that the LRA Rank of best performance increases as we select more parameters to train. In practice, when we tune LIFT with the same number of parameters as LoRA, we find that using LRA Rank similar to LoRA rank yields optimal performance.

References

[1] Meng et al, 2024
[2] Jaiswal et al, 2024
[3] Song et al, 2024