Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

We sincerely appreciate your positive and motivating comments. We are delighted to see that you recognize the novelty and significant performance enhancement of our method. Please kindly see our response to your comment below.

Q1: Regarding the entire LLM, since the proposed method aims to reduce the reconstruction error in layer-wise, will the proposed method progressive prune each layer or jointly prune all the layers for an entire LLM?

A1: Thanks for this insightful question. DS$\oslash$T is proposed as a fine-tuning method to further advance existing sparse LLMs. Instead of fine-tuning weights, we choose a much more efficient alternative, i.e., fine-tuning/editing sparse masks. In our implementation, we progressively apply DS$\oslash$T to fine-tune each layer after pruning. This clarification is now updated to Section 4.1.

Q2: How to assign the sparsity rate for each layer?

A2: As previously elucidated, DS$\oslash$T serves as a fine-tuning methodology utilized to enhance the performance of pruned LLMs. Consequently, our approach does not engage in LLM pruning operations during the initial stage. Instead, we commence by employing established LLM pruning techniques such as magnitude pruning, Wanda, or SparseGPT to initially prune the LLMs. Subsequently, we utilize DS$\oslash$T to further refine the sparse masks obtained from the pruning process.

Therefore, all the configurations of the pruning process strictly adhere to the methodologies outlined in Wanda and SparseGPT. For instance, this involves the allocation of a uniform sparsity rate across all layers as used by SparseGPT and Wanda.

Q3: The diffference when applying DS$\oslash$T to sparsegpt or wanda, can dsnot be an independent method?

A3: We appreciate this question. Indeed, DS$\oslash$T is a versatile and scalable fine-tuning approach applicable to any LLM pruning method as we dicussed before. All we need to do is applying DS$\oslash$T to an established sparse LLM to dynamically prune-and-revive weights by looking at the pruning errors, regardless of the specific pruning method (Wanda, SparseGPT, and Magnitude) employed. DS$\oslash$T is not an independent pruning method for LLMs, rather, it is a highly efficient fine-tuning method for enhancing the performance of sparse LLMs.

Q4: Extra computing and running time compared to Wanda.

A4: You are correct that DS$\oslash$T + Wanda incurs additional overhead costs compared to Wanda. However, we would like to re-iterate that that DS$\oslash$T is designed as a fine-tuning technique for sparse LLMs. It requires merely 4.3 seconds to fine-tune a sparse LLaMA-7B, which is dramatically more efficient than both LoRA fine-tuning (which takes 4 hours as demonstrated in Table 4) and full fine-tuning (requiring several days).

Even though there are some overhead cost when integrating DS$\oslash$T with Wanda, the whole process (including the fine-tuning process) can be finished within merely 4.3s, being much faster even than the pruning only approach, e.g., SparseGPT (using 209s). We humbly suggest that the efficiency of our approach is more a merit than a downside.

We sincerely appreciate the time and diligence you’ve taken to participate in the review of our paper. If you have further questions, we are more than glad to discuss with you.

Dear reviewer nrCN,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards!

Q3: Whether Wanda+DS$\oslash$T can achieve better fully fine-tuned accuracy compared to that of Wanda?

A3: Thank you for this professional inquiry. The answer is yes and DS$\oslash$T is orthogonal to LoRA fine-tuning. As evidenced in the following table, after using DS$\oslash$T to fine-tune sparse LLMs pruned by Wanda, further performance improvements can be achieved by consequently using LoRA fine-tuning. This yields better results than simply fine-tuning the sparse network pruned by Wanda.

• WikiText-2 perplexity performance for using LoRA to fine-tune 50% sparse LLaMA-7B | Sparsity | 0.5 | | ----------------------- | ---- | | Wanda | 7.26 | | Wanda+LoRA | 6.87 | | Wanda+DS$\oslash$T | 7.12 | | Wanda+DS$\oslash$T+LoRA | 6.76 |

Q4: How many random seeds are used throughout the experiments?

A4: We use one random seed in our submission. To demonstrate the robustness of our approach, we have added the results with different calibration sets under five random seeds in Appendix A.2. The variance across random seeds is very low, suggesting the robustness of DS$\oslash$T, corroborating its efficacy as a tool in fine-tuning sparse LLMs. For your convenience, we list the results as below.

• WikiText validation perplexity for pruning LLaMA-V1 and LLaMA-V2 models at 60% sparsity. We report the mean and standard deviation under 5 random seeds. | Method | LLaMA-V1-7B | LLaMA-V1-13B | LLaMA-V2-7B | LLaMA-V2-13B | | --------------- | ----------------- | ---------------- | ----------------- | ---------------- | | Dense | 5.68 ($\pm$0.00) | 5.09 ($\pm$0.00) | 5.47 ($\pm$0.00) | 4.88 ($\pm$0.00) | | SparseGPT | 10.42($\pm$0.04) | 8.43 ($\pm$0.02) | 10.14 ($\pm$0.03) | 7.88 ($\pm$0.01) | | w. DS$\oslash$T | 9.64 ($\pm$0.03) | 7.73 ($\pm$0.02) | 9.68 ($\pm$0.03) | 7.57 ($\pm$0.01) | | Wanda | 10.69 ($\pm$0.01) | 8.75 ($\pm$0.01) | 10.79 ($\pm$0.01) | 8.40 ($\pm$0.01) | | w. DS$\oslash$T | 10.22 ($\pm$0.01) | 8.46 ($\pm$0.01) | 10.59 ($\pm$0.01) | 8.18 ($\pm$0.01) |

Q5: Why is LLM-pruner not included in the baselines while N:M structured pruning is included?

A5: Thanks for your question. The primary goal of this paper is weight pruning, i.e., removing individual weights, including unstructured pruning and n:m sparsity. LLM-pruner itself is a structured pruning, i.e., completely removing entire channels and attention heads, which is not directly comparable to the unstructured pruning approaches. Also due to this reason, both SparseGPT and Wanda do not include LLM-pruner as their baselines.

Your time and effort in reviewing our paper are genuinely appreciated. If there are any additional questions or points that require clarification, we would be more than delighted to engage in further discussions.

We thank the Reviewer s71w for the time and effort to review our paper. We are grateful for the constructive comments. We are glad that the reviewer found our paper to be timely and to be the first work tackling the LLM pruning problem at >50% sparsity regime.

Q1: DS$\oslash$T does not bring the performance of sparse LLMs even remotely close to that of the original dense network. Need reporting experimental results at a lower sparsity regime.

A1: Thank you sincerely for providing this invaluable comment. We would like to express our gratitude for the insights you've shared, which indeed help us to better understand the benefits of our approach.

Firstly, we wish to clarify the primary objective of our paper. Our intention is not to propose an approach that can rival the performance of the original dense LLMs at high sparsity levels. Instead, our focus is on exploring the possibility of further refining pruned LLMs without resorting to the resource-intensive backpropagation, extensive corpus of text, and long training time—conditions that are often impractical in resource-limited scenarios. In contrast to fine-tuning the remaining weights, our paper illustrates an alternative approach: fine-tuning/editing sparse masks to enhance the performance of sparse LLMs. Remarkably, this entire process can be executed with utmost efficiency, utilizing only 25GB GPU memory, 128 sequence of data, 4.3 seconds for pruning LLaMA-7B.

It is worth noting that, as mentioned in Wanda's paper, even after an full fine-tuning process employing 4x A100 GPUs for 3 days, the fine-tuned sparse model still falls short of matching the original performance. Given this context, we acknowledge the reasonable expectation that our approach may encounter similar limitations in matching with the original dense LLM performance, especially at high sparsity.

Moreover, following your suggestion, we have now updated Appendix A.1 to include the results at lower sparsity levels to further showcase the effectiveness of DS$\oslash$T. These results are also listed below for your convenience. Our results indeed show that our approach is able to match the dense performance at mild sparsity levels such as 10%-20%.

• WikiText-2 perplexity performance for fine-tuning LLMs at varying sparsity rates. | Model | Method | 0% | 10% | 20% | 30% | 40% | 50% | | ------------ | --------------- | ----- | ----- | ----- | ----- | ----- | ----- | | LLaMA-V1-7B | Wanda | 5.68 | 5.70 | 5.82 | 6.00 | 6.39 | 7.26 | | LLaMA-V1-7B | w. DS$\oslash$T | 5.68 | 5.68 | 5.73 | 5.89 | 6.28 | 7.12 | | LLaMA-V1-13B | Wanda | 5.09 | 5.10 | 5.13 | 5.25 | 5.51 | 6.15 | | LLaMA-V1-13B | w. DS$\oslash$T | 5.09 | 5.09 | 5.11 | 5.05 | 5.29 | 6.08 | | LLaMA-V2-7B | Wanda | 5.47 | 5.49 | 5.59 | 5.74 | 6.06 | 6.92 | | LLaMA-V2-7B | w. DS$\oslash$T | 5.47 | 5.48 | 5.49 | 5.65 | 5.85 | 6.81 | | LLaMA-V2-13B | Wanda | 4.88 | 4.91 | 4.99 | 5.13 | 5.37 | 7.88 | | LLaMA-V2-13B | w. DS$\oslash$T | 4.88 | 4.89 | 4.91 | 5.01 | 5.25 | 7.57 | | OPT-13B | Wanda | 10.12 | 10.13 | 10.09 | 10.12 | 10.63 | 11.92 | | OPT-13B | w. DS$\oslash$T | 10.12 | 10.12 | 10.08 | 10.11 | 10.41 | 11.28 |

Q2: Trade-off between computational cost and performance improvement

A2: Thank you for this insightful comment. You are correct that DS$\oslash$T + Wanda incurs additional overhead costs compared to Wanda. However, we would like to re-iterate that that DS$\oslash$T is designed as a fine-tuning technique for sparse LLMs. It requires merely 4.3 seconds to fine-tune a sparse LLaMA-7B, which is dramatically more efficient than both LoRA fine-tuning (which takes 4 hours as demonstrated in Table 4) and full fine-tuning (requiring several days).

Even though there are some overhead costs when integrating DS$\oslash$T with Wanda, the whole process (including the fine-tuning process) can be finished within merely 4.3s, being much faster even than the pruning only approach, e.g., SparseGPT (using 209s). We humbly suggest that the efficiency of our approach is more a merit than a downside.

Dear reviewer s71w,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards!

Dear reviewer s71w,

We sincerely thank you for your support. We would like to offer additional clarification regarding the benefits of DS$\oslash$T for hardware-friendly pruning. The current implementation of DS$\oslash$T focuses on pruning and reviving individual weights, making it equally applicable to semi-structured hardware-friendly pruning. As shown in Table 5 of our paper, DS$\oslash$T effectively improves the performance of N:M sparse networks, which achieves notable acceleration supported by the NVIDIA Ampere Sparse Tensor Core[1,2].

As for structured sparsity, completely removing entire channels and attention heads does not directly conform to the current DS$\oslash$T method. However, your point has indeed inspired us to consider adapting DS$\oslash$T to structured sparsity scenarios. One idea is to use the output of the attention block as a guide for the pruning-and-reviving process for entire channels or attention heads, thereby fine-tuning structured sparse LLMs. This will require a more sophisticated design, which is challenging to execute within the limited timeframe of this rebuttal period and beyond the scope of our paper’s current emphasis on weight pruning, i.e., the removal of individual weights. We are committed to leave it as a promising future work building upon this paper. Once again, we are grateful for your supportive feedback and constructive comments.

[1] Nvidia a100 tensor core gpu architecture, 2020. https:// www.nvidia.com/content/dam/enzz/Solutions/Data-Center/ nvidia-ampere-architecture-whitepaper.pdf.

[2] A simple and effective pruning approach for large language models. In Arxiv, 2023.

We sincerely appreciate your careful review, positive feedback, and constructive comments. We are delighted to see that you recognize that we have done a thorough analysis and comprehensive literature reviewing. Please kindly see our responses to your comments below.

Q1: How does the threshold affect the final performance of the model, for both perplexity and efficiency?

A1: Thank you for posing this insightful question. We do agree with your insights and have analyzed the effect of the threshold $\epsilon$ in Section 4.4 (Figure 3) of our submission.

First of all, we would like to highlight that the value of threshold $\epsilon$ does not necessarily lead to too much difference in terms of the computing efficiency, typically resulting in a negligible computing time difference (often less than 1 second), owing to the efficient nature of our approach, as demonstrated in the table below. But indeed, the values of threshold $\epsilon$ has a big impact on the fine-tuning performance. Intuitively, Larger threshold values tend to correlate with increased reconstruction error, potentially leading to suboptimal performance. Conversely, smaller threshold values result in reduced reconstruction error, often associated with enhanced performance. Nonetheless, exceedingly small values may cause overfitting to the limited calibration data, resulting in inferior performance.

The results below verify the above intuition. First, the computing time difference of the whole fine-tuning process is only 0.6 seconds when $\epsilon$ increases from 0 to 1. In terms of performance, perplexity continuously decreases as $\epsilon$ decreases from 1 to 0.1, while their is a slight increase of perplexity when $\epsilon$ becomes too low, e.g., $\epsilon=0$.

• WikiText-2 perplexity performance and running times of DS$\oslash$T for fine-tuning 50% sparse LLaMA-V1-7B pruned by the Wanda metric with different threshold ε | $\epsilon$ | 0 | 0.1 | 0.5 | 1 | | -------------- | ---- | ---- | ---- | ---- | | Computing Time | 4.6s | 4.3s | 4.2s | 4.0s | | PPL | 7.14 | 7.12 | 7.16 | 7.19 |

Q2: The outcomes at 50% sparsity level for comparison.

A2: Thank you for this valuable suggestion. We have also presented the results for pruning LLaMA-V1 with 7B and 65B parameters under a 50% sparsity rate, as can be seen in Table 2. To provide a better overview of our method, we have also included more results at 50% sparsity ratio in Appendix A.1. For your convenience, we list the results below.

• WikiText-2 Perplexity comparison for pruning LLMs at 50% sparsity rate | Method | LLaMA-V1-7B | LLaMA-V1-13B | LLaMA-V2-7B | LLaMA-V2-13B | OPT-13B | | --------------- | ----------- | ------------ | ----------- | ------------ | ------- | | Dense | 5.68 | 5.09 | 5.47 | 4.88 | 10.12 | | Magnitude | 17.29 | 20.21 | 16.03 | 6.83 | 2.96e3 | | w. DS$\oslash$T | 14.04 | 15.52 | 13.09 | 6.31 | 1.10e3 | | SparseGPT | 7.22 | 6.21 | 7.00 | 6.02 | 15.61 | | w. DS$\oslash$T | 7.08 | 6.13 | 6.88 | 5.64 | 14.88 | | Wanda | 7.26 | 6.15 | 6.92 | 5.97 | 11.98 | | w. DS$\oslash$T | 7.12 | 6.08 | 6.85 | 5.87 | 11.68 |

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we would be more than delighted to engage in further discussion with you.

Dear reviewer CfsK,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards!