Dynamic Low-Rank Sparse Adaptation for Large Language Models

We appreciate the time and effort you put in reviewing our work, as well as your detailed comments and valuable questions. We understand the concerns you raised and we're pleased to address these concerns. We also appreciate the opportunity to provide some clarification.

W1: The novelty may be limited.

Answer (1/2): We appreciate the reviewer's feedback and would like to clarify the unique contributions of our work, in contrast to prior methods. Below, we highlight our novel contributions and compare them to the cited works:

1. Comparison of A1-A3: LAMP [A1] proposes an importance score for global pruning, which is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning. This approach is significantly different from our method, which determines layer-wise sparsity rates based on mutual information (MI). A2 proposes retaining neurons in each layer that have high MI values with the neurons retained in the previous layer. This pruning is performed top-down, starting from the softmax layer and progressing to the embedding layer. However, pruning the 1.1B BERT-Extra-Large with A2 requires 16 V100 32GB GPUs, making it impractical for larger models ranging from 7B to 70B parameters. MPruner [A3] uses MI to evaluate inter-layer similarity and clusters layers based on their similarity to identify and remove redundant layers. However, the coarse-grained approach of removing entire layers results in a significant accuracy drop for LLMs, particularly under high pruning rates. Although LoSA, A2, and A3 all utilize MI to determine inter-layer redundancy, LoSA focuses on using MI to determine the layer-wise sparsity rates for sparse LLMs. This is an issue that neither A2 nor A3 addresses and represents a key distinction between LoSA and these approaches.
2. Comparison of A4-A5: LoSA, AdaLoRA [A4], and ALoRA [A5] all focus on developing efficient methods for rank allocation. However, there are significant methodological differences among them. AdaLoRA parameterizes incremental updates using singular value decomposition (SVD) and dynamically adjusts the rank of the incremental matrices based on a designed importance metric. ALoRA evaluates the importance of each rank, iteratively prunes low-contribution ranks, and reallocates resources to achieve dynamic adjustment of ranks. LoSA is a method specifically designed for sparse LLMs, dynamically adjusting parameters by allocating ranks based on the reconstruction error of sparse LLMs. Allocating ranks based on reconstruction error is a novel rank allocation approach proposed in our paper.
3. Comparison of A6-A7: Although our method also uses progressive pruning like [A6, A7], the insight behind our method is that due to the large number of LLMs parameters, pruning all weights at once will cause the reconstruction error of sparse LLMs to increase abnormally, making it difficult to restore the accuracy of sparse LLMs through fine-tuning. Therefore, we extend the one-shot pruning to progressive pruning. This gradual application of pruning effectively mitigates the output deviation caused by one-shot pruning. At the same time, in the process of progressive pruning, we also increase or decrease the rank of low-rank adaptation according to the changes in the reconstruction error of the sparse model, dynamically adjust the fine-tuning parameter allocation to minimize the reconstruction error of sparse LLMs. This approach of using both progressive pruning and dynamic rank allows us to more effectively fine-tune sparse LLMs to improve their performance, which is the unique contribution of our method.
4. Comparison of A7: Although our approach allows the integration of LoRA weights with sparse LLMs like LoRAPrune [A7], it is important to note a key difference: LoRAPrune [A7] is designed for structural pruned LLMs, where LoRA and pruned weights can be seamlessly merged by simply adjusting the input and output dimensions of LoRA. However, this method is not compatible with unstructured pruning, and LoRAPrune cannot be directly extended to fine-tuning unstructured pruned models. We propose simultaneously sparsifying both the low-rank adaptation and the LLMs to preserve the original acceleration advantages of sparse LLMs. Our method can integrate with any training-free unstructured pruning approach. Given the generality and versatility of our approach, we believe that LoSA makes a unique and valuable contribution to the community.

Answer (2/2):

5. Comparison of Zheng et al. (2021): It is important to emphasize that using RMI to determine the layer-wise sparsity rates of LLMs is just one subset of our numerous innovations. Additionally, while we acknowledge that our work is inspired by the methodology presented in Zheng et al. (2021), we believe that our paper introduces significant novel contributions by extending and adapting the RMI from CNNs to LLMs. Zheng et al. (2021) utilized mutual information to determine the layer-wise pruning rates for CNNs, followed by fine-tuning the one-shot pruned CNNs. This work only demonstrated the feasibility of using mutual information as a metric for pruning CNNs. However, due to the significant structural differences between LLMs and CNNs, the application of mutual information for pruning LLMs has not yet been explored. In contrast, LoSA employs a progressive sparse fine-tuning approach to obtain the final sparse LLMs. During this process, the relative importance of each layer in the LLMs changes due to weight updates during fine-tuning. However, given the massive number of parameters in LLMs, determining the relative importance of each layer is both challenging and time-consuming. In this paper, we derive the use of the RMI metric to efficiently and rapidly determine the relative importance of each layer in LLMs during sparse fine-tuning. Through extensive experiments, we validate the effectiveness of the RMI metric in determining sparsity rates for LLMs. Our method is the first to leverage mutual information theory to guide the pruning of LLMs, providing valuable insights to the research community. We acknowledge that the explanation of these unique contributions could be made more explicit in the paper. We will revise the manuscript to better highlight how our work differs from and advances beyond Zheng et al. (2021), emphasizing the innovations specific to LLMs.
6. Contributions of our method: our LoSA method addresses multiple challenges in a single algorithm framework, including determining the sparsity rates of the LLMs, allocating fine-tuning parameters, and resolving the issue of incompatibility between low-rank adaptation weights and sparse LLMs. By providing an end-to-end solution, LoSA effectively tackles various challenges in fine-tuning sparse LLMs. It significantly improves the accuracy of existing sparse LLMs and achieves faster inference compared to LoRA on both CPUs and GPUs. This substantially enhances the deployability of sparse LLMs across different hardware platforms, making a great contribution to the community.

We hope this clarification highlights the uniqueness and impact of LoSA, and we are open to further suggestions to improve the presentation of our contributions. Thank you for your valuable feedback!

W2: The baseline is not enough.

Thank you for your valuable suggestions!

1. Firstly, it should be noted that our paper focuses on fine-tuning unstructured pruned LLMs. SparseGPT and Wanda are popular unstructured pruning methods for LLMs, and LoSA is designed to fine-tune unstructured pruned LLMs obtained by SparseGPT and Wanda. Our paper demonstrates the performance improvements that LoSA brings to these methods. Additionally, LoRA is a highly effective fine-tuning approach and a widely adopted method for fine-tuning unstructured pruned LLMs (the Wanda paper also shows results using LoRA to fine-tune unstructured pruned LLMs). Therefore, we believe that comparing our LoSA method with the LoRA baseline is reasonable and enough.
2. Secondly, based on your suggestions, we extended the LoSA method to fine-tune structured pruned LLMs and used the wanda-sp method for pruning. We compared LoSA with SliceGPT, LLM-Pruner, and LoRAPrune on the LLaMA-7B model with 20% pruning rate. The experimental results are as follows:

Our LoSA method outperforms SliceGPT, LLM-Pruner, and LoRAPrune as shown in the table, demonstrating the superior performance of LoSA. We have also included the experimental data on fine-tuning structured pruned LLMs in the revised version. Thank you again for your valuable suggestions.

W3: The setting with 2:8 or mixed 2:8 sparsity may not be solid.

Thank you for your valuable suggestions!

1. Since mixed 2:8 sparsity is a specific type of unstructured sparsity pattern, it can also leverage the nm-vllm sparse inference engine to achieve accelerated inference on GPUs. The acceleration effect is as follows:

2. Secondly, a major contribution of our paper is determining the layer-wise sparsity rates for LLMs. For N:M sparsity, we use our RMI-based sparsity rates method to determine the N value in mixed N:M sparsity. However, for 2:4 sparsity, our RMI-based sparsity rates method does not work.
3. Additionally, based on your suggestions, we also conducted experiments on 2:4 sparsity and mixed 2:4 sparsity on LLaMA2-7B obtained by Wanda method. For 2:4 sparsity, we do not use the RMI-based sparsity rates method. In contrast, for mixed 2:4 sparsity, we apply the RMI-based sparsity rates method. The experimental results are as follows.

LoSA improves accuracy for both 2:4 sparsity and mixed 2:4 sparsity and outperforms LoRA. We also present experimental data showing the use of the nm-vllm sparse inference engine to accelerate inference on GPUs for 2:4 sparsity and mixed 2:4 sparsity models. Since both 2:4 sparsity and Mixed 2:4 sparsity have a sparsity rate of 50%, the acceleration effects are essentially the same.

W4: Comparison with Sparse LoRA

Thank you for your suggestions. We present the comparison of accuracy and inference acceleration between Sparse LoRA, LoRA and LoSA. Specifically, we show the results for the 70% sparse LLaMA-2-7B model.

Accuracy comparison:

Inference acceleration comparison:

Since Sparse LoRA's Low-rank adaptation is also sparse, its accuracy is worse than LoRA, and it is also much worse than LoSA. Additionally, since both Sparse LoRA and LoSA can be merged into sparse LLMs, their inference acceleration effects are basically the same, and both outperform LoRA. Overall, our proposed LoSA method outperforms both Sparse LoRA and LoRA in terms of accuracy and inference acceleration.

Finally, I hope that my response has addressed your concerns, and I kindly ask for a more favorable evaluation of our work. Thank you!

Dear Reviewer VoNN,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards,

Authors

Dear Reviewer VoNN,

We have put in many efforts to address the raised questions and concerns. As the ICLR rebuttal period is approaching its end, we kindly remind you to review our submitted response. Additionally, we sincerely hope that you can reconsider the score of our paper. Your feedback is essential for finalizing our work.

Thank you for your attention.

Best regards,

Submission417 Authors

Thanks for the rebuttal. After going through the rebuttal, I still have some concerns for the novelty. As I mentioned in my review, this paper proposes three improvements including dynamic sparsity rates across layers with RMI, dynamic rank allocation, and progressive pruning, which have been explored and demonstrated effect in LLM pruning. I mentioned some related works and I know they are different from this paper. I list these papers to show that the ideas in this paper have been explored in pruning of DL or LLM models.

Furthermore, some technical components in the paper are almost the same as the previous works. The RMI method in this paper is almost the same as that in [Zheng et al. (2021)]. The problem formulation, algorithm design and RMI definition based on HSIC are exactly the same. The authors may change the expressions but they are indeed the same. The progressive pruning method of this paper is also the same as the progressive sparsity in [A6] for LLM pruning.

The rebuttal highlights other things. [Zheng et al. (2021)] uses RMI to determine the layer-wise sparsity for convolutional layers, and this paper uses RMI to determine the layer-wise sparsity for linear layers. I do not see there are too many differences here. Convolutional layers are specific forms of linear layers, and this step just determines the sparsity rates. How to prune under the given sparsity rates are determined by other methods suitable for CNN or LLM. It does not seem to be a significant contribution to determine the sparsity rates for linear layers compared with convolutional layers using the same method. The RMI method relies on the output of the layers, without touching the unique computation patterns of linear or convolutional layers.

We greatly appreciate your time, detailed comments, and valuable suggestions. We are pleased to address your concerns and make some clarifications.

W1 & W2: This paper ignores previous works [1-4] on sparse PEFT using Low-Rank Adaptation.

Thank you for your valuable suggestions!

1. Firstly, it should be noted that our paper focuses on fine-tuning unstructured pruned LLMs. SparseGPT and Wanda are popular unstructured pruning methods for LLMs. Our paper demonstrates the performance improvements that LoSA brings to these methods. Additionally, LoRA is a highly effective fine-tuning approach and a widely adopted method for fine-tuning unstructured pruned LLMs (the Wanda paper also shows results using LoRA to fine-tune unstructured pruned LLMs). Therefore, We compare our LoSA with LoRA.
2. Secondly, SoRA [1] is to dynamically adjust the rank of low-rank adaptation in the training process with a sparse gating unit trained by proximal gradient method. The only similarity between LoSA and SoRA lies in their adaptive rank adjustment in low-rank adaptation during fine-tuning. However, there are significant differences between the two methods. LoSA dynamically allocates ranks to each layer based on reconstruction error. Moreover, LoSA can be merged into sparse LLMs, whereas SoRA cannot. Additionally, LoSA also introduces a method for determining layer-wise sparsity rates for LLMs based on mutual information and adopts an iterative sparse fine-tuning approach. In the revised version, we have added a discussion of SoRA in the related work section. Based on your suggestion, we compared the SoRA and LoSA methods:

The results show that LoSA method outperforms SoRA. This is evident because SoRA only dynamically adjust the rank. In contrast, LoSA dynamically adjusts the rank based on reconstruction error, determines layer-wise sparsity rates for sparse LLMs, and adopts iterative sparse fine-tuning. These strategies ensure that LoSA achieves better accuracy than SoRA.

3. Thirdly, both LoRAPrune [2] and LoRAShear [3] fine-tune structured pruned LLMs. We have discussed related content in the related work section. In these works, low-rank adaptation is dense and cannot be merged into unstructured pruned LLMs. Consequently, this paper aims to explore effective techniques for integrating low-rank adaptations into the unstructured pruned LLMs. The goal is to ensure that sparse LLMs and low-rank adaptations share the same sparse mask, thereby the model’s sparsity is preserved and inference latency remains unaffected. Since our paper focuses on fine-tuning unstructured pruned LLMs, we do not include them in the comparison baselines. According to your suggestion, we extended the LoSA method to fine-tune the structured pruned LLMs, and used the wanda-sp method to obtain the LLaMA-7B with 20% pruning rate. We compared the LoRAPrune and LoRAShear. The experimental data of LoRAPrune and LoRAShear come from LoRAShear paper.

LoSA outperforms LoRAPrune and LoRAShear.

4. Finally, LoSparse [4] performs structured pruning on smaller models like BERT, using a combination of low-rank and sparse matrices to approximate the original weight matrix. LoSparse involves structured pruning of the sparse matrix, followed by training the low-rank matrix to compensate for the sparse matrix, resulting in structured pruned BERT. The only similarity between LoSA and LoSparse lies in their use of iterative pruning. However, there are significant differences between two methods. LoSA introduces innovations such as sparsity alignment of LoRA and sparse LLMs weights, determination of layer-wise sparsity rates for LLMs, and adaptive adjustment of the rank in LoRA during fine-tuning. In our revised version, we added a discussion on LoSparse in the related work section. Since the LoSparse method does not present experimental results on LLMs, and LoSparse requires the same memory as full model fine-tuning, we lack sufficient GPU resources to demonstrate LoSparse results on models larger than 7B. As a result, it is difficult to directly compare the advantages and disadvantages of LoSA and LoSparse.

We have included all experimental data mentioned above in the revised version. Thank you again for your valuable suggestions.

W3: The author could also compare with some latest works of LLM post-training pruning works.

Thank you for your suggestions. It should be noted that our LoSA method is designed for fine-tuning unstructured pruned LLMs, which means that LoSA can be combined with any LLM post-training unstructured pruning methods to enhance their accuracy. We have demonstrated the improvement that LoSA brings to SparseGPT and Wanda, both of which are popular post-training unstructured pruning methods for LLMs. In the table below, we further show the performance improvements of LoSA on the latest LLM post-training pruning methods, including Pruner-Zero [A1] and ALPS [A2]. All experimental data are based on a 70% sparse LLaMA2-7B model.

From the data in the table, we can observe that the accuracy of post-training pruned LLMs has significantly decreased compared to the dense model. LoSA effectively improves the accuracy of post-training pruned LLMs, outperforming LoRA. We have also included the above experimental data in the revised version. If you would like us to show the fine-tuning effects of LoSA on other LLM post-training pruning works, please let us know.

[A1] Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models. ICML 2024.

[A2] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. NeurIPS 2024.

W4: The improvements over baselines are incremental, some of them are very marginal.

We thank you for your comment and appreciate the opportunity to clarify our contributions:

1. Our paper’s experiments cover multiple architectures (LLaMA, Vicuna, OPT) and sizes, presenting results across different sparsity ratios. Performance improvements in both perplexity (Table 1), zero-shot accuracy (Table 2) and sppedup (Table 9) underscore LoSA’s adaptability across models and sparsity ratios. Experimental results show that LoSA has a significant improvement over the two training-free sparsity methods including Wanda and SprseGPT. For example, LoSA reduced the perplexity of 70% sparse LLaMA-2-7B using Wanda method by 68.73 and increased zero-shot accuracy by 16.32%, achieving a 2.60$\times$ speedup on CPU and 2.23$\times$ speedup on GPU. In the above case, LoSA has 1.63 lower perplexity than LoRA, 3.47% higher zero-shot accuracy, 0.31$\times$ faster speedup on CPU, and 0.52$\times$ faster speedup on GPU. This is a considerable improvement.
2. Additionally, the accuracy degradation of sparse LLMs obtained by SparseGPT and Wanda method is relatively small compared to dense models at lower sparsity rates. Thus, there is limited room for improvement in model accuracy through fine-tuning at low sparsity rates. As a result, LoSA offers less improvement for LLMs with low sparsity rates than that with high sparsity rates. However, LoSA further reduces the accuracy gap between low sparsity LLMs and dense LLMs. For instance, the zero-shot accuracy of a 50% sparse LLaMA2-7B model obtained through SparseGPT shows a gap of 2.59% compared to the dense model, while LoSA reduces this gap to 0.56%, outperforming the 0.85% achieved by LoRA method. Furthermore, LoSA enables the creation of a lossless 60% sparse LLaMA2-70B model, achieving a 0.84% higher accuracy compared to the LoRA method. In contrast, a 60% sparse LLaMA2-70B model obtained by SparseGPT exhibits a 2.87% accuracy drop relative to the dense model. The precision-lossless sparse LLMs achieve inference acceleration on both CPU and GPU, which significantly enhances practical deployability and once again proves the superiority of our method.

Thank you for your consideration, and we look forward to your feedback.

Finally, I hope that my response has addressed your concerns, and I kindly ask for a more favorable evaluation of our work. Thank you!

Dear Reviewer 7hsM,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards,

Authors

Dear Reviewer 7hsM,

We have put in many efforts to address the raised questions and concerns. As the ICLR rebuttal period is approaching its end, we kindly remind you to review our submitted response. Additionally, we sincerely hope that you can reconsider the score of our paper. Your feedback is essential for finalizing our work.

Thank you for your attention.

Best regards,

Submission417 Authors

Dear Reviewer 7hsM,

As the deadline of the discussion phase is approaching, we kindly ask you to review our response to see whether it can address your concerns. Following your suggestions, we have compared LoSA with related works [1-4], demonstrated the novelty of our approach, and show that LoSA improves on the latest LLM post-training pruning works.

If our response has addressed your concerns, would you mind reconsidering your score?

Thank you once again for your effort and valuable feedback.

Q4: It would be helpful if authors could also elaborate on the computational overhead introduced by the RMI and reconstruction error-based adaptations. Specifically, how does this additional computation scale with model size?

Thank you for your suggestion. We present the computation overhead for RMI-based sparsity and reconstruction error-based adaptations for the 7B to 70B models in the table below. All data is collected on a single NVIDIA A100 80GB GPU.

We can observe that as the model size increases, the required computation time also increases. However, for the largest 70B model, the computation time for RMI-based sparsity and reconstruction error-based adaptations are only 332 seconds and 321 seconds, respectively, which are very fast and have minimal computational overhead.

Q5: Could authors also clarify the interpretability of RMI as a sparsity allocation metric? While the Information Bottleneck principle supports its use, it would be helpful to see additional justification or experiments demonstrating that RMI consistently aligns with real layer importance (e.g., gradient-based layer importance) across diverse architectures.

Thank you for your suggestion. To the best of our knowledge, there is currently no work on gradient-based layer importance for LLMs, so we cannot compare the layer-wise sparsity rates obtained by the RMI metric with such approaches. If you have gradient-based layer importance methods for determining LLMs layer-wise sparsity rates, please let us know and we would be happy to add comparisons with these methods.

We plot a line graph of layer-wise sparsity rates for different LLMs in Figure 3 of our paper. We observed that the RMI metric tends to assign lower sparsity rates to the initial and final layers of LLMs while allocating higher sparsity rates to the middle layers. This can be explained as follows: the initial layers of LLMs are typically responsible for processing low-level features of the input, which are crucial for the subsequent layers' understanding. The middle layers primarily handle information processing and feature extraction, where some redundancy often exists. The final layers focus on generating high-quality contextual representations, directly influencing the model's output. This sparsity allocation strategy is similar to the approach adopted by LLM-Pruner [1], which does not prune the first three layers or the last layer but prunes the middle layers. It also aligns with the findings of SLEB [2], which prunes LLMs by discarding unimportant layers. According to Table 4 in the SLEB paper, the discarded layers are mainly concentrated in the middle parts of the LLMs.

[1] LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023.

[2] SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. ICML 2024.

Q6: Given that LoSA’s speedups are measured using specific sparse inference engines (e.g., DeepSparse, nm-vllm), how generalizable are these results to other deployment environments?

Thank you for your questions.

1. First, due to the sparsity of weights in unstructured pruning, we must use a specific sparse inference engine to accelerate inference. We use DeepSparse and nm-vllm to accelerate inference on general deployment environments, including CPU and GPU.
2. For CPUs, the DeepSparse tool supports architectures including: x86 AVX2, AVX-512, AVX-512 VNNI, and ARM v8.2+, which covers most Intel, AMD, and Apple M-series CPUs.
3. For GPUs, as long as the GPU supports CUDA installation, inference acceleration can be achieved using the nm-vllm tool. Similarly, if CUDA installation is supported for other deployment environments, nm-vllm can also be used to achieve acceleration. If you have any questions about other specific deployment environments, please feel free to let us know.

Finally, I hope that my response has addressed your concerns, and I kindly ask for a more favorable evaluation of our work. Thank you!

We appreciate the time and effort you put in reviewing our work, as well as your detailed comments and valuable questions. We understand the concerns you raised and we're pleased to address these concerns.

W1&Q1: Could authors provide results comparing LoSA with adaptive LoRA baselines like AdaLoRA and SoRA?

Thank you for your suggestions. We present comparison of LoSA with AdaLoRA [1] and SoRA [2] below. Since AdaLoRA and SoRA have only been experimented on smaller models (such as BERT) and there is no experimental data for LLMs (such as LLaMA), we use the open source codes of AdaLoRA and SoRA to fine-tune 70% sparse LLaMA2-7B obtained by the Wanda. Since the rank of LoRA is 8, according to the original paper, the initial rank for each incremental matrix in AdaLoRA is 12. The rank of the SoRA method is set to 8, and other hyperparameters are set according to the original paper. Other experimental settings follow those in Section 3.1 and are aligned with the settings of LoRA and LoSA.

The results show that LoSA outperforms AdaLoRA and SoRA. This is evident because AdaLoRA and SoRA only dynamically adjust the rank, and the weights of AdaLoRA and SoRA cannot be merged into sparse LLMs. In contrast, LoSA dynamically adjusts the rank based on reconstruction error, determines layer-wise sparsity rates for sparse LLMs, and adopts dynamic sparse fine-tuning. Additionally, LoSA weights can be merged into sparse LLMs. These strategies ensure that LoSA achieves better accuracy than AdaLoRA and SoRA. We have also included the experimental data in the revised version.

W2&Q2: In Eq. 5, why is the sparsity rate set to increase with the layer importance score?

We apologize for any confusion caused during your reading, and your understanding is correct. We assign lower sparsity rates to the more important layers. We define the sparsity rate $s_i$ as the ratio of zero elements to the total number of elements in each layer. Consequently, Equation (5) is updated to $\textcolor{red}\min_{\boldsymbol{s}} \boldsymbol{p}^T\boldsymbol{s} \, s.t. \frac{1}{n}\sum_{i=1}^{n}s_i = {\Theta}$, changing the original maximization objective into a minimization one. We have corrected the above errors in the revised version of the manuscript, and we sincerely apologize again for any inconvenience caused by these oversights.

W3&Q3: How does LoSA ensure consistent sparsity patterns between the LoRA (BA) and LLM (W) weights?

This issue can be illustrated with reference to Figure 1 and Equation 1 from the paper, which provide an intuitive explanation. Specifically, post-training sparsification methods like SparseGPT and Wanda essentially aim to derive a mask $\boldsymbol{M}$ for the weights $\boldsymbol{W}$ of the LLM. In the original LoRA, the $\boldsymbol{BA}$ matrices are dense. Consequently, the expression $\boldsymbol{M}\odot \boldsymbol{W} + \boldsymbol{BA}$ cannot achieve sparsity consistency. LoSA addresses this by also sparsifying the $\boldsymbol{BA}$ matrices, applying the same mask $\boldsymbol{M}$ as the LLM weights $\boldsymbol{W}$. This results in $\boldsymbol{M}\odot (\boldsymbol{W} + \boldsymbol{BA})$, thereby achieving sparsity consistency between the LLM weights $\boldsymbol{W}$ and the LoRA $\boldsymbol{BA}$ matrices.

W4: Further analysis is needed on the soundness and effectiveness of the RMI-based sparsity and reconstruction-based rank allocation across different architectures.

Thank you for your suggestions. We further conducted ablation experiments on the OPT model, which is a non-LLaMA architecture, using RMI-based sparsity and reconstruction-based rank allocation. Specifically, we used the Wanda method to obtain a 70% sparse OPT-13B. The results are as follows:

We can see that removing either the RMI-based sparsity or the reconstruction-based rank allocation leads to a decrease in LoSA accuracy. The results demonstrate the soundness and effectiveness of the RMI-based sparsity and reconstruction-based rank allocation across different architectures.

Dear Reviewer MFoH,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards,

Authors

Thanks for your careful review and comments! Hopefully the following contents could answer your questions.

W1: the algorithm consists of many heuristics and is lack of step by step derivation, e.g., Eq. 7 and Eq. 9

We thank the reviewer for pointing out this issue. In this paper, the determination of sparsity rates for LLMs is derived from the Information Bottleneck theory, rather than relying on heuristic designs. For the low-rank adaptation, we indeed employ heuristic approaches to derive Eq. 7 and Eq. 9. However, the rank of the matrices serves as a hyperparameter, making it challenging to determine the rank for LoRA through theoretical derivations. Many existing studies [1–3] aim to gradually adjust the rank during the fine-tuning process to achieve an optimal rank allocation for LoRA. This paper advocates for the dynamic allocation of the rank parameter budget during sparse LLMs finetuning process, based on the sparse reconstruction errors across different layers. Therefore, we designed Eq. 7 and Eq. 9. Furthermore, while our rank determination is based on heuristic formulas, the ablation studies in Table 5 validate the effectiveness of our adaptive rank adjustment for sparse LLMs. In the future, we will try to prove the validity of Eq. 7 and Eq. 9 with rigorous mathematical proofs.

[1] AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning.

[2] Sparse low-rank adaptation of pre-trained language models.

[3] ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models.

W2: some experiment details are missing and unclear.

Thank you for your valuable feedback. Perhaps the experiment details you are talking about are mentioned in the question section? We will answer questions about the experiment details in the question section. If you feel that other experiment details are missing and unclear, please explain in more detail.

Q1：section "RMI for Sparsity" with Eq. 2 and Eq. 3 are not used in the proposed algorithm which can confuse readers for their purpose. this paper could shorten this presentation.

Thank you for your suggestions. In the section "RMI for Sparsity," we derived the RMI metric to determine the importance of each layer in LLMs. While Eq. 2 and Eq. 3 are not directly used in the proposed algorithm, they ultimately lead to the derivation of Eq. 4 and Eq. 5. Eq. 4 and Eq. 5 provide the formulas for determining layer importance and calculating sparsity rates. Therefore, we included this section in the paper to present the complete derivation. We have streamlined the expressions in this section as much as possible. If you have any further questions or concerns regarding the presentation of this section, please let us know.

Q2: Eq. 7 has a hidden assumption that higher rank setting can help reduce reconstruction loss. According to Eq. 1, under current problem formulation, higher rank does not change anything given the sparsity mask M. Thus, the design of Eq. 7 is not useful. Similarly, this question also applies to Eq. 9. Why Eq. 9 is performed at each step and what happens if Eq. 9 became too large?

1. When the rank $r$ of matrices $\boldsymbol{B} \in \mathbb{R}^{c_{cout}\times r}$ and $\boldsymbol{A} \in \mathbb{R}^{r \times c_{in}}$ increases, the parameters of both $\boldsymbol{B}$ and $\boldsymbol{A}$ in Eq. 1 are increased. After fine-tuning, $\boldsymbol{B}$ and $\boldsymbol{A}$ can better compensate for the reconstruction error $\mathcal{L}$ between the sparse and dense model.
2. Additionally, the average rank of all layers is set to $\Omega$ in Eq. 7. Under this constraint, we allocate a larger rank to layers with higher reconstruction error, as these layers will require more fine-tuning parameters to reduce the error effectively.
3. Because we adopt a progressively increasing sparsity strategy, the sparsity rate gradually grows, which leads to an increase in the reconstruction error of the sparse model. To address this issue, we increase the average rank of all layers as the sparsity step increases in Eq. 9. Therefore, both Eq. 7 and 9 are valid. We investigate the impact of rank on the perplexity of the final model in Figure 2. Increasing the rank appropriately can effectively reduce perplexity, leading to a better recovery of the sparse model’s performance. However, since the fine-tuning samples are fixed, further increasing the rank results in insufficient training of low-rank adaptation, which causes an increase in perplexity.

Q3: It is unclear how rank increase and decrease is implemented in practice. For example, if current rank is 3 and you want to increase to 4, do you initialize the additional vector randomly? if current rank is 4 and you want to decrease to 3, do you perform rank reduction by singular values magnitude (if you use svd)?

1. For training stability, when two low-rank matrices $\boldsymbol{B} \in \mathbb{R}^{c_{out}\times r}, \boldsymbol{A} \in \mathbb{R}^{r \times c_{in}}$ increase their rank, we concatenate random Gaussian initialized parameters $\mathcal{N} (0, \sigma^2)$ to $\boldsymbol{A}$ and zeros to $\boldsymbol{B}$. The above initialization operation is the same as LoRA, so the layer’s output remains unchanged before and after new parameters added.
2. When two low-rank matrices $\boldsymbol{B}, \boldsymbol{A}$ decrease their rank, the converted $\boldsymbol{B}^{'}, \boldsymbol{A}^{'}$ directly inherit parameters from the original matrix $\boldsymbol{B}, \boldsymbol{A}$ and the extra parameters are discarded.

Q4: It should be discussed the computation complexity. In particular, the sparsity mask computation can be slow. Given the algorithm is executed at each fine-tuning step, the overall computation time should be reported.

We have discussed the computation complexity in Section 3.6 and shown the overall computation time in Table 8. Specifically, since LoSA uses multiple sparsification operations and needs to obtain sparse masks multiple times, the overall computation time of LoSA is longer than that of LoRA. Nevertheless, LoSA provides better accuracy and lower inference latency compared to LoRA, and it only requires about an hour of fine-tuning, which we believe is a worthwhile trade-off. If you have any questions about our analysis, please let us know.

Q5:Experiment result table report "SparseGPT with LoSA", while the Algorithm 1 input is dense weight of LLM. Which part of dense weight of SparseGPT you work on?

In Algorithm 1, we start with a dense LLM weight as input, and SparseGPT determines the sparsity mask for the LLM, which is then applied to the dense weight to obtain a sparse weight. The LoSA method fine-tunes the sparse LLM obtained by SparseGPT to improve the accuracy of the sparse LLM.

Q6: Which layer of LLM and weight matrix (Q,K,V, etc) you run experiment with?

We apply sparsity to all Linear layers in the Attention and MLP modules following the setup of the Wanda and SparseGPT methods. For the LLaMA model, this includes sparsifying the weight matrices of Q, K, V, Out Proj, Up Proj, Down Proj, and Gate Proj, while excluding the final LM Head. Additionally, we add low-rank adaptation to all sparse weight matrices. In Table 11, we present an ablation study that shows the effect of applying low-rank adaptation to only certain sparse weight matrices (e.g., Q, K or Q, K, V, O). The experimental results demonstrate that applying low-rank adaptation to all sparse weight matrices yields the best performance.

Q7:What is the outcome of SparseGPT with LoRA? Does it become a dense LLM? If so, what does sparsity 50% mean for SparseGPT with LoRA?

Since the LLM weights are sparse and the LoRA weights are dense, merging the dense LoRA weights into the sparse LLM weights would cause the LLM weights to become dense as well. Therefore, for a sparse LLM, the LoRA branch can only be retained without merging, in order to preserve the compression and acceleration benefits of the sparse LLM. However, retaining the LoRA branch increases the inference time of the sparse LLM. In contrast, the LoSA weights are also sparse and can be merged into the sparse LLM weights, avoiding any increase in inference time for the sparse LLM. For a comparison of inference time between LoRA and LoSA, please refer to Table 9.

Q8: Is there any guideline for setting fine-tuning steps? In Table 1, T=5, why 5 steps can be sufficient for completing LoRA training?

We have presented the ablation study of the fine-tuning steps T in Figure 2 and discussed the impact of different fine-tuning steps on the final perplexity in Section 3.5. Steps T determines the frequency at which the sparsity rate increases. A larger T means the sparsity rate increases more slowly, and fewer parameters are removed each time. For each fine-tuning step, we first apply sparsity to the LLM, and then fine-tune the sparse LLMs using 2K Alpaca-GPT4 samples. Since we use a sufficient number of fine-tuning samples, this is enough to train LoRA. We show the impact of different values of T on the final perplexity while keeping the number of fine-tuning samples constant. Increasing T appropriately can effectively reduce the model’s perplexity. However, a larger T may result in insufficient training of the model after each sparsification, which in turn leads to a further increase in perplexity.

Finally, I hope that my response has addressed your concerns, and I kindly ask for a more favorable evaluation of our work. Thank you!

Dear Reviewer FSq8,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards,

Authors

We greatly appreciate your time, detailed comments, and valuable suggestions. Below, I will address each of your questions one by one.

Q1：Can you comment (qualitiatively) on the applicability of the mutual information based layer-sensitivity method to other compression techniques -- e.g., would it work for quantization, or if we were jointly quantizing and sparsifying?

1. We determine the importance of each layer based on the mutual information metric, assigning corresponding sparsity rates to each layer. Layers deemed more important are allocated lower sparsity rates. Similarly, we can also determine the importance of each layer using the mutual information metric and assign an appropriate number of quantization bits to each layer for quantization, with more important layers allocated higher bit precision.
2. Likewise, we can use the mutual information metric to determine the compression rate for each layer in a jointly quantized and sparse model. Specifically, based on the layer importance derived from the mutual information metric, more important layers are assigned lower compression rates. Based on the obtained compression rate, we can further determine the sparsity rate and quantization bits of each layer.

Q2: You have a brief section on N:M sparsity, where you fix M=8. How easy is it to extend this to also determine the right value of M for different layers? Relatedly, any notes on smaller values of M (like M=4, found in Nvidia GPUs as 2:4 sparsity)?

1. We extend LoSA to N:M sparsity and adopt a mixed N:8 sparsity (N refers to non-zero weights) configuration, and M value is fixed for different layers, you can set it to any value (such as 8 or 4). Meanwhile, we allow different layers to have distinct N values while maintaining a constant overall sparsity ratio. We assign lower N values to less important layers and the N value for each layer are determined using the mutual information based sparsity method. Since mixed 2:8 sparsity is a specific type of unstructured sparsity pattern, it can also leverage the nm-vllm sparse inference engine to achieve accelerated inference on GPUs. The acceleration effect is as follows:

2. Secondly, we can also easily use a smaller value of M to obtain a mixed 2:4 sparsity model. We conducted experiments of mixed 2:4 sparsity on LLaMA2-7B obtained by Wanda method. The experimental results are as follows:

LoSA improves accuracy for mixed 2:4 sparsity and outperforms LoRA. We also present experimental data showing the use of the nm-vllm sparse inference engine to accelerate inference on GPUs for LLaMA2-7B with mixed 2:4 sparsity.

Q3: Can you add more color to the cost of your proposed method during training (i.e., impact on training time). There is an annotation that it takes 48 seconds for LLama-2-7B. Can you clarify: (i) what is the relative contribution to the overall step time, (ii) if the computations are performed in every step (or less frequently like every k steps -- or if it is possible to do it less frequently).

1. The layer-wise sparsity rates for the LLaMA-2-7B can be determined in an end-to-end time of only 48 seconds on a single NVIDIA A100 80GB GPU using our method. Moreover, since we adopt a progressive sparsification approach, we use a frequency T = 5, where the sparsity rate increases from 0 to the target sparsity rate. Therefore, determining the layer-wise sparsity rates takes a total of 48 seconds $\times$ 5 = 4 minutes. However, the total fine-tuning time for the LLaMA-2-7B is 45 minutes, making the time spent on sparsity rate computation relatively few compared to the overall fine-tuning time.
2. Of course, we can adjust the sparsification frequency T, either increasing or decreasing it, which will proportionally increase or decrease the time required for sparsity rate computation. Additionally, in Figure 2, we investigate the impact of different sparsification frequency T on the final perplexity of the model. Increasing T appropriately can effectively reduce the model’s perplexity. However, a larger T may result in insufficient training of the model after each sparsification, which in turn leads to a further increase in perplexity.

Finally, I hope that my response has addressed your concerns. Thank you!

Thank you for your responses. It addresses most of my questions.

I think your response to [Q3] can use a little more details. I understand the overall (end-to-end) overhead of your method on training time. Would it be possible to annotate the specifics I requested: (i) what is the overhead of performing your method relative to an individual step, (ii) how frequently is it performed?

For example, if your training run is: 100k steps @ 0.05 ms per step := 5000 seconds (total) + your sparsity approach is applied every 20k steps and takes 10 seconds when applied := 50 seconds (total). Thus total runtime is 5050 seconds, of which 50 seconds is spent in your sparsity approach. Then: (i): Sparsity approach takes 50 seconds vs. 0.05 ms per step. (ii): The sparsification is done 5 times across the entire training duration.

(A): I am inferring the 5 times in (ii) is the same as T=5 in your terminology. Correct? (B): I am inferring the 50 seconds in (i) is the same as 48 seconds in your terminology. Correct? (C): If you can annotate either: the time per step (the equivalent of 0.05 ms), or total steps (100k), the reader can derive the remainder.

I am interested in (i) simply to understand what is the relative overhead, and because for a practitioner it can dictate how they may be willing to set the frequency in (ii). Or alternatively, it can pinpoint a computational bottleneck that the systems community can work on alleviating (as future work). As a different example, second-order optimizers are promising but expensive, so we either run them infrequently, or improve systems support for the relevant operations. I think it would be helpful for readers to understand the same calculus for your approach.

Thank you again for your explanations and good work. My decision was originally Accept, and I will retain this score.