LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning

Q1. This paper can benefit from better writing and presentation.

Thanks for your suggestion. We have carefully checked and modified the equations and presentation, and reflected these changes in the revised paper.

Q2. Could you present any supporting experimental results on how accurate LoRA-guided importance approximations are?

A2. We have already provided the results and discussions in "how accurate LoRA-guided importance approximations are" in the alation study. We set the vanilla gradient-guided criterion in Eq. (3) as our baseline, since our criterion is an efficient approximation of vanilla criterion. As shown in Figure 4, the similarity between these two criteria is very high with Ratio = 10%, which indicates that our LoRA-guided criterion can accurately estimate the importance in an iterative way.

Q3. Does a better approximation yield better performance eventually?

A3. It's noteworthy that, compared to the vanilla criterion in Eq. (3), the LoRA-guided criterion is designed for more efficient evaluation of weight importance, not necessarily for enhanced performance. Evidence of this has been presented in Table 10 (Table 5 in the initial submission) and Table 4, which illustrates that while the LoRA-guided criterion maintains performance parity with the vanilla criterion, it significantly reduces the fine-tuning time by 63.6% and GPU memory usage during pruning by 52.6%. This underscores the effectiveness of our method in achieving high-performance levels more efficiently.

Q4. It seems $\hat{I}$ has the same dimension as the model weights W?

A4. Yes, $\hat{I}$ has the same dimension as the model weights $W \in R^{d\times k}$. However, we do not need to save $\hat{I}$ for all layers in each iteration. We can calculate $\hat{I}$ group by group. For example, we simultaneously calculate importance of Query, Key and Value in $g$-th Attention layer. Subsequently, we accumulate group importance $\mathcal{\hat{G}}_g \in R^{1\times heads}$ according to Eq. (4) and only $\mathcal{\hat{G}}_g$ needs to be saved, where $heads << d\times k$.

Q5. It would be helpful for authors to provide ‘theoretical FLOPs’ compared to dL/dW-based importance criterion.

A5. We provide the theoretical FLOPs between LoRA-guided criterion and vanilla gradient-guided (dL/dW-based) criterion in the following table. The FLOPs computation rule can be referred to [1]. The table shows LoRA-guided criterion has lower theoretical FLOPs than Vanilla, as it does not compute pretrained weight gradients. We add this comparison in Table 8 in the revised paper.

[1] Scaling laws for neural language models. Arxiv 2020

We sincerely thank you again for your great efforts in reviewing this paper. We have addressed your major concerns about the presentation and provided theoretical FLOPs compared to the dL/dW-based importance criterion. Please don’t hesitate to let us know if there are still concerns/questions.

Q1. This paper lacks the motivation why the LoRA can considered as the guidance. It is encouraged to illustrate why this criterion is created.

A1. As explained in the introduction, our primary motivation for employing the LoRA-guided criterion is to reduce computational and memory demands during iterative pruning. As depicted in Figure 1(b), the conventional gradient-guided criterion depends on the gradients of tpre-trained weights, necessitating the computation and storage of gradients with a shape of $d\times k$. In stark contrast, the LoRA-guided criterion requires only the gradients of matrices $A$ and $B$ with shapes of $r\times k$ and $d\times r$, respectively, where $r$ is significantly smaller than $d$ and $k$ (often by a factor of a thousand, as in the case of LLaMA-7B with $d=4096$ and $r=8$). This substantial reduction in size translates to a corresponding reduction in computational load and memory usage. The efficacy of the LoRA-guided criterion, as demonstrated by our experimental results in Figure 4, is its capability to accurately gauge the importance of each weight group, mirroring the accuracy of the traditional gradient-guided approach but with much greater efficiency.

We have incorporated the analysis into the METHOD section in the revised paper.

Q2. The equations in this paper are unclear and not easy to understand, as there are a lot of abnormal superscripts and subscripts.

A2. We have explained the superscripts and subscripts in General Response Q3. We have reflected these changes in the revised paper.

We sincerely thank you again for your great efforts in reviewing this paper. We have addressed your major concerns about motivation and presentation. Please don’t hesitate to let us know if there are still concerns/questions.

Dear Reviewer C5GK,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Q1. Table 2 shows a significant degradation in Perplexity (PPL).

A1. We thoroughly address this concern in our General Response Q1. It is critical to highlight that our method significantly outperforms both LLM-Pruner and WANDA at token segments of 128 and 2048.

Notably, our LoRAPrune is complementary to large-scale fine-tuning to mitigate the performance drop, by simply replacing the small post-training calibration data with the large-scale dataset for fine-tuning. Following Sheared-LLaMA [4], we use RedPajama corpus (400k data) to iteratively prune LLaMA-7B. The experimental results can be found in our General Response Q1.

Q2. Difference with LLM-Pruner.

A2. We have discussed the difference between LoRAPrune and LLM-Pruner in General Response Q2.

Q3. I have reservations about the pivotal significance of the 'starting point' for fine-tuning, especially when the pruned model is meticulously fine-tuned.

A3. We first re-emphasize the practical significance of post-training pruning with limited calibration data:

1. Acquiring large volumes of fine-tuning data for Large Language Models (LLMs) is often infeasible.

2. The computational burden of fine-tuning LLMs on extensive datasets is substantial, which can limit their practical deployment.

Given these constraints, our research pivots towards post-training pruning using minimal calibration data. In this setting, the iterative pruning is vital for achieving high-compression rates in models—a strategy that has garnered considerable attention in previous pruning literature [1,2,3]. For instance, as presented in Table 6, the 'Vanilla' model—which can be viewed as an iterative variant of LLM-Pruner—demonstrates a significant reduction in PPL when fine-tuned iteratively, down to 29.96 from 38.12, highlighting the efficacy of iterative pruning.

Furthermore, in response to the reviewer’s query, we further include meticulous fine-tuning scenarios. We evaluate the effectiveness of iterative pruning on large-scale dataset (RedPajama corpus [5]) and compare it with one-shot pruning. The experimental results are shown in the following table. It is noteworthy that even with such data volume (400k), iterative pruning still outperforms one-shot pruning markedly. We include these results in Table 4 in the revised paper.

[1] Unified and progressive pruning for compressing vision-language transformers. Arxiv 2023

[2] Width & depth pruning for vision transformers. AAAI 2022

[3] Deep Compression of Pre-trained Transformer Models. NeurIPS 2022

[4] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. Arxiv 2023

[5] Redpajama: An open source recipe to reproduce llama training dataset. 2023

Q4. The memory constraints of LLM-Pruner can be effectively mitigated through off-loading.

A4. Gradients off-loading results in reduced throughput which can be found in General Response Q2.

We sincerely thank you again for your great efforts in reviewing this paper. We have addressed your major concerns about significant PPL degradation and the necessity of iterative pruning. Please don’t hesitate to let us know if there are still concerns/questions.

Q4. I question the practical significance of 'post-training calibration with limited data.

A. Fine-tuning the pruned models on a large-scale dataset needs prohibitive computing resources, which can be infeasible for research communities with limited resources. For example, Sheared LLaMA [1] uses RedPajama corpus with 50B tokens and costs about 120 GPU days for fine-tuning. Consequently, much work in the compression community concentrates on post-training scenarios [2-4], utilizing only minimal calibration data for pruning and performance restoration.

Besides, large-scale fine-tuning is not viable in privacy-sensitive or data-limited settings, which are frequently encountered in many real-world scenarios such as in medical, finance, and autopilot domains, where obtaining the data is often challenging due to privacy or confidentiality issues. Therefore, there is a huge demand in industry and academia for pruning LLMs with limited data.

Furthermore, our experiments with datasets of 30k and 400k demonstrate that LoRAPrune's performance can be enhanced by scaling up the fine-tuning dataset. Hence, our approach is not constrained by the size of the dataset. In the future, we plan to extend LoRAPrune to a larger RedPajama corpus to further boost its performance.

[1] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. Arxiv, 2023

[2] Brecq: Pushing the limit of post-training quantization by block reconstruction. ICLR 2021

[3] A fast post-training pruning framework for transformers. NeurIPS 2022

[4] Bayesian bits: Unifying quantization and pruning. NeurIPS 2020

Q5. The reduction in memory and runtime during the pruning stage seems inconsequential when considering the significantly longer duration of the fine-tuning stage.

A. Since we use iterative pruning and the moving average technique, the importance of weights must be calculated for each batch during fine-tuning. To clearly delineate the time spent on pruning versus fine-tuning, we provide the GPU hours on the c4 corpus (30k) for LLM-Pruner with iterative pruning and LoRAPrune in the following table, based on LLaMA-7B. Pruning time consists of the time spent on computing the importance, mask adjustment and, if needed, gradients of pre-trained weights. In addition, fine-tuning time covers the forward pass, gradients computation and updates of LoRA weights.

The table indicates that LoRAPrune spends only 10% of the total time on pruning, while LLM-Pruner with iterative pruning spends 66% of the time on pruning.

Q1. PPL degradation in Table 2. Such a pronounced PPL drop would be deemed acceptable.

A1. We thoroughly address this concern in General Response Q1. It is critical to highlight that our method significantly outperforms both LLM-Pruner and WANDA at token lengths of 2048 and 128, as shown in Table 2 and Table 7, respectively. In scenarios where a pronounced PPL drop is observed, it is often possible to mitigate this through the use of additional data to refine the pruning results.

Q2. The results from WANDA also indicate a significant PPL degradation.

A2. The WANDA study primarily reported on unstructured and semi-structured pruning results. Our focus, as discussed in our paper's introduction, is on structured pruning due to the hardware efficiency limitations associated with unstructured sparse models. To ensure a fair comparison, we replicated the structured pruning results using WANDA's official codebase with necessary modifications. We make our adapted code available in the supplementary materials to strengthen the reproducibility of our results. Consistent with the aim of fairness, we employed WANDA's metrics to calculate the weight importance, subsequently aggregating the group importance as delineated in Equation (4). The specific modifications are documented in 'wanda/lib/prune.py' lines 171-193 for further examination and reproducibility.

Q3. The 50% pruning rate appears to present substantial challenges.

A3.In the post-training structured pruning context, especially when limited calibration data is available, a 50% pruning rate indeed presents substantial challenges. However, we significantly outperform other structured pruning methods. To mitigate the PPL degradation in 50% ratio, we also iteratively prune LLMs on large-scale dataset. The details can be found in General Response Q1.

Q4. It would be beneficial to have the PPL and other metrics as reported by contemporaneous studies.

A4. We have duly noted the request for comprehensive metrics. In response, Table 2 has reported PPL and accuracy metrics for zero-shot task classification on the common sense reasoning dataset. To further validate the efficacy of LoRAPrune, we have supplemented these with results from the MMLU (5-shot) evaluations, offering a broader performance perspective consistent with contemporaneous studies. We include these results in Table 5 in the revised paper.

We sincerely thank you again for your great efforts in reviewing this paper. We have addressed your major concerns about significant PPL degradation. Please don’t hesitate to let us know if there are still concerns/questions.

Dear Reviewer nuRr,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Q1. The novelty of this paper is limited.

A1. We propose the innovative LoRA-guided pruning criteria, which is informed by low-rank adaptation (LoRA) rather than relying on the gradients of pre-trained weights. This efficient criteria enables us to structurally prune 65B LLMs in a single GPU, improving the applicability of pruning on LLMs. We discuss the differences between LoRAPrune and LLM-Pruner in General Response Q2. Reviewers nuRr and RcVc agree with our novelty.

Q2. Does LLM-Pruner not support tuning by LoRA?

A2. We apologize for any confusion caused. To clarify, LLM-Pruner supports tuning with LoRA but does not support iterative pruning efficiently. Specifically, LLM-Pruner uses one-shot pruning and then fine-tunes the pruned models by LoRA. The reasons for LLM-Pruner's inefficiency with iterative pruning are discussed in our General Response Q2. The term "Fine-tune" as referenced in Table 1 pertains to the process of iterative pruning which involves simultaneous pruning and fine-tuning to regain performance. To clarify this process and avoid further misunderstanding, we have updated the terminology in Table 1 from "Fine-tune" to "Iterative Pruning" in our revised submission.

Q3. The author missed an important baseline, LLM-Pruner, in more large scale models.

A3. We acknowledge the significance of including LLM-Pruner as a baseline for larger models. Due to the limited timeframe of the rebuttal period, we provide results for the 50% pruning ratio. These results are detailed in the table below.

The preliminary results, as shown in above table, indicate that LoRAPrune outperforms LLM-Pruner in both the LLaMA-13B and LLaMA-30B models at a 50% pruning ratio. This reinforces the scalability and effectiveness of our proposed LoRAPrune. We include these results in Figure 3 in the revised paper.

We sincerely thank you again for your great efforts in reviewing this paper. We have addressed your major concerns about novelty and additional comparison with LLM-Pruner. Please don’t hesitate to let us know if there are still concerns/questions.

Dear Reviewer pQXa,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Q2. Difference from LLM-Pruner. (Reviewers pQXa, nG8N)

A2. We are the first method that unifies efficient fine-tuning with structured pruning. Our method has fundamental differences with LLM-Pruner [1]:

1. Our work is the first to integrate LoRA with iterative structured pruning, achieving both parameter-efficient tuning and direct hardware acceleration during inference. In particular, our new LoRA-guided pruning criteria uniquely leverages only LoRA's weights and gradients, making the pruning process memory-efficient and fast. Therefore, LoRAPrune enables scalability to LLaMA-65B on a single GPU. This is in sharp contrast to LLM-Pruner, which uses full gradients for pruning and thus hard to generalize to large-scale models under limited computational and storage resources. The experimental results in Table 2 and Table 8 demonstrate that LoRAPrune outperforms LLM-Pruner by a perplexity reduction of 4.81 on WikiText2 and 3.46 on PTB datasets, while concurrently reducing memory usage by 52.6%.

2. Although LLM-Pruner’s memory demands can be partially mitigated by off-loading strategies, such as transferring certain gradients to CPU memory, the memory access cost and computational overhead are substantial. The following table clearly illustrates the efficiency gains achieved by our method in LLaMA-7B. In terms of throughput, LoRAPrune is 2.75$\times$ faster than LLM-Pruner with offloading and 8.19$\times$ faster without offloading. This enables us to use iterative pruning to mitigate the drop arising from structured sparsity. We include these results in Table 8 in the revised paper.

Overall, LoRAPrune represents a fundamental advancement over LLM-Pruner, both in technical methodology and in the broader implications of its application.

[1] LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023

[2] A Simple and Effective Pruning Approach for Large Language Models. Arxiv 2023

[3] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023

[4] Redpajama: An open source recipe to reproduce llama training dataset. 2023

Q3. This paper can benefit from better writing and presentation.

A3. We list the main modifications as follows:

We polish the notations following the suggestions from Reviewer RcVc. For instance, to avoid confusion with the use of subscripts and superscripts, we revised Equation (4) and now use $\mathcal{G}$ to represent what was originally $\mathbf{I}_g$. Additionally, we outline the notations employed in the equations at the beginning of Section 3.1.