Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

Thanks for your careful review and comments!

Weaknesses (Smaller improvements for larger models.)

We have observed that larger models show less performance improvement with ATP. This observation aligns with common patterns in model pruning and compression, where the returns tend to diminish as model size increases. However, our ATP still demonstrates consistent improvements across different model sizes, especially notable in larger models:

1. Significance of Relative Gains

While absolute improvements may appear smaller for larger models, the relative gains remain significant. For example, for LLaMA-65B at 70% sparsity with Wanda, ATP reduces the zero-shot accuracy loss from 10.81% to 6.65%, representing a 38.48% relative improvement. However, the improvement of AlphaPruning is only 30.71%. Such gains can translate to meaningful performance enhancements in real-world applications.

2. Inherent Resilience of Larger Models

Larger models naturally exhibit greater resistance to pruning. As shown in Table 1, 70% sparse LLaMA2-70B with SparseGPT experiences an zero-shot accuracy loss of only 8.30%, compared to 18.81% for LLaMA2-7B, leaving less room for further improvement. Nevertheless, our method further reduces the loss of LLaMA2-70B to 6.89%. This observation aligns with the findings of Li et al. [1].

3. Alignment with Theoretical Insights

The behavior aligns with recent theoretical insights. Liu et al. [2] demonstrated that larger models can maintain performance even under random pruning, supporting the idea of their inherent pruning resistance.

In summary, our ATP method remains effective across all model sizes and still achieves considerable performance improvements in larger models, providing critical stability and performance benefits.

[1] Li et al. Train big, then compress: Rethinking model size for efficient training and inference of transformers.

[2] Liu et al. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.

Other Comments Or Suggestions (Figure 1 needs to be enlarged.)

Thank you for your suggestion. We will enlarge Figure 1 in our final version to make it clearer for viewing.

Question 1 (What are the advantages of sparsity compared to quantization?)

Network sparsity and quantization are both effective methods for improving inference speed and reducing memory footprints, with both techniques offering significant acceleration benefits across different hardware platforms (such as CPU and GPU). However, comparisons between these two approaches in terms of efficiency may vary.

Recent research suggests that network sparsity can slightly outperform quantization in inference speedup. For example, SqueezeLLM [3] demonstrates that quantizing LLaMA2-7B to 3 bit achieves a 2.1$\times$ speedup on GPU. Meanwhile, the latest sparse CPU and GPU kernels (DeepSparse and nm-vllm) provide better support for deploying sparse LLMs, achieving 2.63$\times$ CPU inference acceleration and 2.23$\times$ GPU inference acceleration for 70% sparse LLaMA2-7B, as shown in Table 6 of our paper. Additionally, compared to quantization, sparsity can better maintain and recover performance through fine-tuning, as described in [4].

Pruning and quantization are compatible and complementary approaches, combining these methods can further enhance efficiency. Table 3 in [4] shows that combining network sparsity with quantization (INT8) can achieve up to 9.08$\times$ acceleration, significantly higher than using either method alone.

[3] SqueezeLLM: Dense-and-Sparse Quantization.

[4] Sparse Fine-tuning for Inference Acceleration of Large Language Models.

Question 2 (Experimental results with 60% sparsity.)

We present the zero-shot accuracy results of Llama-3-8B at 60% sparsity below. The sparse model is obtained using the Wanda method, and we compare ATP with other layer-wise sparsity methods.

We observe that:

1. ATP significantly improves the accuracy of Wanda, increasing the average zero-shot accuracy by 4.13% and narrowing the performance gap between sparse and dense model.

2. ATP outperforms OWL, DSA, and AlphaPruning, with 1.05% higher accuracy than the best-performing AlphaPruning.

In conclusion, our ATP method demonstrates consistent improvements across various sparsity levels and significantly outperforms existing layer-wise sparsity methods.

We will incorporate the above responses into the final version. We hope that our response has addressed your concerns. Thank you!

Thanks for your careful review and comments!

Weakness 1 (Smaller improvements for lower sparsity.)

As sparsity decreases, the returns on performance improvements tend to diminish, which is consistent with common patterns in model pruning and compression. However, we believe that our ATP method still demonstrates quite impressive performance:

1. Significance of Relative Gains

While absolute improvements may appear smaller for models with lower sparsity rates, the relative gains remain significant. For 50% sparse LLaMA-7B obtained using Wanda, our ATP method reduced the average zero-shot accuracy loss from 6.03% to 3.37%, showing a 44.11% relative improvement, compared to only 30.85% relative improvement with OWL.

2. Inherent Capabilities of Lower Sparsity Models

At lower sparsity settings, the inherent capabilities of the model are better preserved. For a 50% sparse LLaMA2-7B obtained using Wanda, the average zero-shot accuracy loss is only 3.66%, compared to 26.55% at 70% sparsity, leaving less room for further improvement.

3. Alignment with Theoretical Insights

Wang et al. [1] established scaling laws for sparse LLMs, revealing the relationship between sparsity rate and model performance, indicating that model performance loss is smaller at lower sparsity rates.

Overall, our ATP method maintains effectiveness across all sparsity settings, offering critical stability and performance advantages.

[1] Wang et al. Q-Sparse: All Large Language Models can be Fully Sparsely-Activated.

Weakness 2 (Implementation details of CNN, ViT and LLaVA-1.5.)

1. Following Wanda, we only sparsify the linear layers within each block of the ConvNeXt, and we use our ATP method to determine the sparsity rate for each block.

2. For ViT, we use ATP to determine the sparsity rate for each layer. Each layer contains modules such as attention and MLP. We use Wanda to sparsify linear layers.

3. For LLaVA-1.5, we only sparsify the Vicuna model within it. Similar to LLaMA, we determine the sparsity rate for each layer and sparsify linear layers.

Other Comments Or Suggestions (Typos.)

Thanks for this great and detailed comments. We will fix this typo in the final revision.

Question 1 (Lack of MoE experiments.)

We use Wanda to obtain 70% sparse MoE model Mixtral-8x7B.

Our ATP method performs excellently on sparse MoE models, further demonstrating its generalizability across different model architectures.

Question 2 (Lack of Llama-3.2-1B/3B experiments.)

We use SparseGPT to obtain 70% sparse Llama-3.2-1B/3B.

Our ATP method demonstrates consistent improvements across all parameter levels, performing well for models ranging from 1B to 70B parameters.

Question 3 (On which devices can sparse LLMs be deployed?)

1. Due to the sparsity of weights in unstructured pruning, we must use a specific sparse inference engine to accelerate inference. We use DeepSparse and nm-vllm to accelerate inference on general deployment environments, including CPUs and GPUs.
2. For CPUs, DeepSparse supports architectures including: x86 AVX2, AVX-512, AVX-512 VNNI, and ARM v8.2+, which covers most Intel, AMD, and Apple M-series CPUs.
3. For GPUs, as long as the device supports CUDA, inference acceleration can be achieved using the nm-vllm. Similarly, if CUDA is supported for other deployment environments, nm-vllm can also be used to achieve acceleration.

We will incorporate the above responses into the final version. We hope this addresses your concerns. Thank you!

Thanks for your careful review and comments!

Weakness 1 (Missing sparsity preserving fine-tuning results.)

Thank you for your suggestions. The results of fine-tuning 70% sparse LLaMA2-7B obtained by Wanda using LoSA [1], LoRS [2] and SPP [3] are below.

We observe that:

1. After Fine-tuning, ATP still Outperforms other Methods

After fine-tuning using sparsity-preserving methods, the advantages of ATP have been further highlighted. After fine-tuning with LoSA, the perplexity of ATP is 2.00 lower than Wanda and 0.81 lower than AlphaPruning. In terms of average zero-shot accuracy, after fine-tuning with LoSA, ATP achieves 2.44% higher than Wanda and 1.01% higher than AlphaPruning.

We believe that since ATP better preserves the performance of the one-shot pruned model, the performance of the sparse model is easier to recover through fine-tuning in this case. Therefore, the layer-wise sparsity rate determined by ATP still retains its advantages after fine-tuning with the sparsity-preserving fine-tuning methods and remains the optimal layer-wise sparsity.

2. Significance of Relative Gains

While absolute improvements may appear smaller for fine-tuned models, the relative gains remain significant. For example, compared to dense model, Wanda w. LoSA shows an average zero-shot accuracy loss of 12.09%, while ATP w. LoSA shows a loss of 9.65%, representing a relative improvement of 20.18%. In contrast, AlphaPruning w. LoSA only achieves a relative improvement of 11.83%. Considering that the performance gap between sparse and dense LLMs further decreases after fine-tuning, the aforementioned accuracy improvement is likely to translate into more meaningful performance enhancements in practical applications.

We will include these results in our final version.

Weakness 2 (Performance of the sparse model with 70% sparsity has a gap compared to dense model.)

We have observed that performance of 70% sparse LLMs obtained by ATP still underperform compared to dense models. This observation is consistent with the common pattern in model pruning and compression, where high sparsity rates lead to significant accuracy degradation. However, we believe our ATP method still offers the following advantages:

1. Pushing the Performance Limits of Sparse LLMs

Our ATP method addresses the challenge of performance degradation under high sparsity rates, advancing the frontier of sparse LLMs performance and narrowing the gap with dense models, which is beneficial to the community.

2. Good Performance at Lower Sparsity Rates

Under lower sparsity settings, our ATP method further reduces the performance gap between sparse and dense LLMs. For example, the average zero-shot accuracy loss for 50% sparse LLaMA2-13B model obtained using the Wanda method is 3.52%, while our ATP method reduces this loss to 1.07%. This is highly advantageous for the practical deployment of lower-sparsity LLMs.

3. Further Narrowing the Gap with Dense Models When Combined with Fine-Tuning

As demonstrated in Table 16 and response to Weakness 1, we show that combining ATP with PEFT techniques can further improve the accuracy of sparse LLMs. For example, the average zero-shot accuracy loss for 70% sparse LLaMA2-7B model obtained using Wanda is 26.55%. With ATP and LoSA optimizations, this accuracy loss is further reduced to 9.65%. Moreover, sparse LLMs obtained using the ATP method achieve higher accuracy, retaining this advantage even after fine-tuning.

In summary, although there remains a gap between sparse and dense LLMs under high sparsity settings, our ATP method significantly narrows this gap, greatly enhancing the practicality of sparse LLMs.

Finally, we hope our response has addressed your concerns. Thank you!

Thanks for your careful review and comments!

Theoretical Claims (Any assumptions should be included in the text of Theorem, not just appear in the proof.)

We will restate Theorem 3.1 as: When the input is the same, increasing the sparsity of the weights in the $i$-th layer will lead to an increase in the reconstruction error of this layer.

Experimental Designs Or Analyses (Report average and variance of results.)

Following OWL, DSA and AlphaPruning, all experimental results are conducted under a single fixed random seed. We have reported WikiText2 perplexity of 70% sparse LLaMA2-7B obtained by Wanda across five random seeds and different calibration sets. The variance across random seeds is very low, suggesting the robustness of ATP.

Other Comments Or Suggestions (How do we determine that the $β$ range is actually "small"?)

1. In Sec. 3.3, we state that the reasonable range of $β$ is $0<β≤0.019$ for 70% sparse LLaMA3-8B. To find the best $β$, we use grid search with step size of 0.002, needing only 9 searches. We also test smaller step sizes in Table 4 but find no improvement in results. The step size of 0.002 balances search efficiency and performance well. Therefore, we claim that the reasonable range for $β$ is small, where "small" refers to the limited number of searches required.
2. We analyze the impact of $β$ on perplexity of sparse model in Figure 4, finding that $β$ significantly affects performance. Figure 2 shows that for lower average sparsity, smaller $β$ values are optimal.

Question 1 (What is the relevance of theoretical analysis on linear networks when the focus is on transformers?)

In Sec. 3.1, we represent a layer's computation as $\boldsymbol{WX}$, where $\boldsymbol{W}$ is the layer's weight and $\boldsymbol{X}$ is input. A layer includes components such as attention, nonlinearities, and layer normalization. We acknowledge that there are differences between the theoretical analysis based on $\boldsymbol{WX}$ and the actual architecture, given the presence of more complex nonlinear computations in the network. However, we believe our analysis remains reasonable for the following reasons:

1. The theoretical modeling of $\boldsymbol{WX}$ is sufficient to analyze the layer's reconstruction error.

Our method sparsifies the linear layers in Attention and MLP modules, while other components remain unaffected. These linear layers account for the majority of the parameter count and significantly influence the computation results of the layer. The sparsified linear layers dominate the computation of the reconstruction error for each layer. Although various nonlinear operations exist in the actual architecture, they typically do not fundamentally alter the reconstruction error of each layer and have minimal impact on theoretical analysis. Therefore, we believe modeling the primary computation of a layer as $\boldsymbol{WX}$ is sufficient for analyzing the reconstruction error of that layer. This is also sufficient for us to analyze how reconstruction errors accumulate and propagate across the network.

2. Modeling the computation of modules as $\boldsymbol{WX}$ is a common practice in many works.

[A1] and [A2] simplify the computation of the CONV+BatchNorm+RELU modules in quantized convolutional neural networks as $\boldsymbol{WX}$ when analyzing reconstruction error. This approach of ignoring unnecessary computations and focusing on the core computations is a common practice, which facilitates the derivation of theoretical results.

Thank you for providing valuable insights, which are very important for improving our work. We will include the above clarification into the final version.

[A1] Up or down? adaptive rounding for post-training quantization. ICML 2020.

[A2] Solving oscillation problem in post-training quantization through a theoretical perspective. CVPR 2023.

Question 2 (Discuss Frankle et al.'s work [1])

Thank you for providing such awesome work!

1. Frankle et al. [1] suggest that the effectiveness of initialization pruning mainly depends on each layer's pruning rate, rather than the specific selection of weights within layers. This highlights the importance of layer-wise sparsity rates and supports the value of our work.
2. We have discussed and compared various layer-wise sparsity methods, including those for LLMs (methods in Table 1) and CNN/ViT (methods in Table 14). Unlike all previous methods, our ATP method discovers that using a simple monotonically increasing arithmetic progression for layer-wise sparsity can achieve excellent results.

We will include Frankle et al.'s work [1] in our final version.

We will include the above discussions in our final version. We hope this response has addressed your concerns and kindly ask for a more positive evaluation of our work. Thank you!