Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment

Dear Reviewer 2MGm,

Thank you so much for the detailed and constructive comments, and the recognition of the novelty of the proposed method, the writing, and the experimental evaluation on image classification benchmarks. Please see our responses below to your concerns and questions one by one.

Q1. Overlap between introduction and related work sections:

Our response:

We appreciate the reviewer's keen observation regarding the overlap between our introduction and related work sections.

To address this issue and enhance the overall coherence of our manuscript, we will do the following reorganization:

(1). Introduction: We'll streamline this section to focus on:

• Presenting the core problem of LLM compression
• Outlining our motivation for developing a new approach
• Providing a high-level overview of our key contributions and approach

(2). Related Work: Correspondingly, the Related Work section will be expanded and refined to:

• Offer a more in-depth analysis of existing methods
• Clearly distinguish our approach from previous work
• Provide a critical evaluation of the strengths and limitations of current techniques

By implementing these changes, we aim to improve the logical flow of the paper, eliminate redundancy without sacrificing important content, and offer readers a clearer roadmap of the field and our contributions to it.

Q2. Lack of details on normalization and reweighting strategies

Our response:

Thank you for highlighting the need for more details on our normalization and reweighting strategies.

To address this concern, we'll add a new subsection titled "Normalization and Reweighting Strategies" to our methodology section. This addition will include:

(1). Feature Normalization: We'll detail our approach to standardizing input features across layers, explaining how this helps balance the influence of different feature scales. For example, we'll expand on the process mentioned in Section 3.2, line 185, which introduces feature normalization.

(2). Weight Normalization: We'll describe our process for normalizing layer weights, which is key to ensuring fair comparisons across the network. We'll elaborate on the statement in Section 3.3, starting from line 220.

(3). Reweighting Mechanism: We'll introduce our novel reweighting strategy that adjusts the importance of layers based on their position and contribution to the model's output.

(4). Integration with ALS: Finally, we'll explain how these techniques are seamlessly integrated into the broader ALS framework, demonstrating how they enhance its overall effectiveness.

By including this information, we aim to provide a clearer, more comprehensive picture of our approach.

We appreciate your attention to detail, as it allows us to present a more robust and transparent description of our research.

Dear Reviewer 7kYG,

Thank you for your valuable and insightful comments. We have provided detailed responses to your concerns and questions below.

Q1. Pruned models' performance compared to dense models

Our response:

(1). Uncommon and Unfair Comparison in Sparsity Research

• Comparing sparse models to dense models of similar size is uncommon in sparsity research. Typically, sparse large models are compared to sparse smaller models, as comparing them to unpruned models is generally considered unfair.
• Sparse models underperforming similarly-sized dense models is common with SOTA methods like Wanda[41], SparseGPT[13], and Magnitude[26] pruning. This is not specific to our approach, but rather a characteristic of current sparsification techniques. This comparison is not the primary focus of our work.
• Obtaining and post-training sparse large models is cost-effective using pre-trained models. For example, 4 GPU hours in A100 80GB of LoRA training achieve better performance and faster inference than dense models. Training a dense model like Llama2 7B from scratch requires 184,320 GPU hours.

(2). Performance Improvements and Efficiency Gains

• Tables 2 and 3 in the paper show that our ALS method consistently improves performance across various model sizes and families at 50% sparsity. Notable examples include:

  (a). LLaMA-V3 8B:

  ​ Magnitude pruning: Perplexity reduces by1069 (from 1.1e3 to 30.20)

  ​ SparseGPT: Accuracy increases by 9.82%

  (b). LLaMA-V2 13B:

  ​ SparseGPT: Perplexity reduces by 1.88

  ​ Wanda: Accuracy increases by 2.1%

• Further improvements with LoRA fine-tuning (Table C and D in the gobal rebuttal):

  (a). LLaMA-V1 30B (50% sparse) compared to 13B dense:

  ​ Perplexity: Reduced by 1.21

  ​ Accuracy: Increased by 2.94%

  (b). LLaMA-V2 13B (50% sparse) compared to 7B dense:

  ​ Perplexity: Reduced by 0.13

  ​ Accuracy: Increased by 0.53%

• Inference acceleration:

  (a). LLaMA-V2 7B at 50% sparsity: 2.355x speedup in inference time

  (b). LLaMA-V2 13B (50% sparse) vs 7B dense: Better throughput (0.0118 vs 0.0084 items/sec), Lower latency (41.24 vs 58.20 ms)

• Significant decrease in memory usage, at 50% sparsity, 47.5% memory reduction as shown in Table F in the global rebuttal.

Q2&Q3. Intra-layer pruning calculation method and Integration of ALS with baseline methods

Our response:

The integration of ALS with baselines involves three main steps, ensuring a fine-grained and adaptive sparsity distribution at both inter-layer and intra-layer levels:

(1). Calculate adaptive sparsity ratios for blocks

• Compute the Redundancy Metric (RM) matrix for layers (Equation 4)
• Apply Linear Optimization (Section 3.3)

(2). Calculate adaptive sparsity ratios for sub-blocks

• Compute RM matrix for intra-layer components
• Use Linear Optimization for optimal intra-layer sparsity

(3). Apply sparsity ratios to baseline pruning methods

• Use the calculated ratios as input for existing pruning techniques

This approach ensures a hierarchical, adaptive sparsity distribution throughout the network architecture. The detailed algorithm pipeline is illustrated in Figure 1 of the paper

Note: Detailed implementation is in the submitted code, to be released upon paper acceptance

Q4. Meaning of L-i+1 in equation (5)

Our response:

To clarify the meaning of L - i + 1 in equation (5), it's important to note that this term plays a crucial role in calculating each layer's average independence with subsequent layers:

Calculation method: For layer i, we compute its independence with layers i+1 to L, as shown in Appendix B.3, line 595.

Examples:

(1). Layer 2: average over L-1 layers (divided by L-1). Subsequent layers follow this pattern

Benefits of this approach:

This formulation captures the hierarchical nature of neural networks and provides adaptive sparsity allocation. Adaptive weighting prioritizes earlier layers, ensuring independence from previous layers and reflecting each layer's importance.

Q5. Discrepancy in analysis of 13B vs 7B model

Our response:

Thank you for identifying this typo, which should refer to the 7B model. We will fix this typo in the revision.

Q6 & Limitation. Effect of combining Wanda+ALS with LoRA fine-tuning

Our response:

In addition to the LoRA experiment presented in Table 5 of the paper, we have conducted further experiments. The effect of combining Wanda+ALS with LoRA fine-tuning copmaring with Wanda+ALS is significantly positive (Table C and D in the global rebuttal).

(1). For LLaMA-V1 at 50% sparsity: 30B have PPL reduced by 1.43 and ACC rose by 2.09%; 13B have PPL reduced by 1.15 and ACC rise by 1.89%; 7B have PPL reduced by 4.82 and ACC rose by 10.17%.

(2). For LLaMA-V2: 13B have PPL reduced by 1.56 and ACC rose by 1.23%; 7B have PPL reduced by 1.72 and ACC rose by 2.21%.

Notably, LoRA fine-tuning uses 2000 C4 samples (same dataset as Llama pretraining) in a zero-shot setting, unrelated to evaluation tasks.

Q7. Relationship between sparsity and efficiency

Our response:

Our study shows significant improvements in computation time and memory usage with sparsity:

(1). Computation Time & Memory Usage, Table E,F in the global rebuttal:

• At 50% sparsity: 2.355x speedup, 47.5% memory reduction
• At 70% sparsity: 2.729x speedup, 63.9% memory reduction

(2). Practical benefits: Even at moderate sparsity (50%), significant improvements in both speed and memory are achieved

In conclusion, our sparsification approach significantly reduces computation time and memory usage without substantial performance loss, while also supporting 2:4 and 4:8 structured pruning (see Appendix C).

Thank you for your detailed response, which has resolved most of my concerns. Therefore, I am raising my score to 5.

Dear Reviewer ZqVF,

Thank you very much for your detailed and constructive feedback. We will address your concerns and questions as follows.

Q1. Marginal improvement at low sparsity ratios

Our Response:

At lower sparsity levels (20-30%), improvements from ALS are less pronounced, as even uniform pruning performs well when only a small number of parameters are removed. For example, with LLaMA-V2-7B at 20% sparsity in Table 2 in the paper, perplexity only improves from 8.92 to 8.90 using Wanda w ALS (dense is 8.71), which leave very small room to improve.

However, our method maintains model stability across various sparsity levels and model sizes, showing consistent improvements. It particularly excels at high sparsities crucial for real-world deployment, accelerating inference in resource-constrained environments. For instance, with the LLaMA-V2-13B model (Table 1):

(1). At 50% sparsity: Perplexity reduces by 1.88

(2). At 60% sparsity: Perplexity reduces by 4.91

(3). At 70% sparsity: Perplexity dramatically reduces from 1.4e3 to 204.17

These results demonstrate ALS's effectiveness, particularly in preventing model collapse. Specifically:

(1). LLaMA-V2-7B at 60% sparsity: 83.23 perplexity where Magnitude pruning fails.

(2). LLaMA-V2-13B at 70% sparsity: 204.17 perplexity where Magnitude pruning fails.

Llama3 70B in Table A in the global rebuttal, shows that:

(1). At 50% sparsity: Perplexity improves from 8.49 to 7.24 with SparseGPT+ALS

(2). At 60% sparsity: 13.01 to 12.36

ALS excels in scenarios requiring substantial model size reduction, offering a versatile tool for efficient LLM deployment across various compression levels. Consequently, these results underscore our approach's effectiveness, highlighting its potential for efficient deployment in resource-limited environments.

Q2. Decreased improvement for stronger models

Our response :

We have observed that larger models show less performance improvement with our ALS method. This observation aligns with common patterns in model pruning and compression, where returns tend to diminish as models grow in size. However, our ALS method still shows consistent improvements across different model sizes, especially notable in larger models and at higher sparsity levels:

(1). Significance of Relative Gains

While absolute improvements may appear smaller for larger models, the relative gains remain substantial. For Llama3 70B at 50% sparsity, our ALS method improves perplexity from 8.49 to 7.24 with SparseGPT, representing a 14.72% relative improvement. In the context of large language models, such gains can translate to meaningful performance enhancements in real-world applications.

(2). Inherent Capabilities of Larger Models

Larger models exhibit a natural resistance to pruning. As shown in Table 2 of the paper, the LLaMA-V1 65B model at 50% sparsity only increases in perplexity from 4.93 to 7.37 with Wanda, leaving small room for improvement. Nevertheless, our method further reduces this to 7.15. This observation aligns with the findings of Li et al. [1].

(3). Effectiveness at Higher Sparsity Levels

Our method demonstrates particular strength at higher sparsity levels, especially for the Llama3 70B model, as shown in Tables A and B in the global rebuttal. For instance:

• At 70% sparsity, Wanda w. ALS reduces PPL by 32.98 compared to Wanda alone.

• At 60% sparsity, Magnitude w. ALS reduces PPL by 341.16 compared to Magnitude alone.

These significant improvements in performance retention at high sparsity levels are crucial for practical deployments where substantial model size reduction is required.

(4). Alignment with Theoretical Insights

The behavior we observe aligns with recent theoretical insights. Liu et al. [2] demonstrated that larger models can maintain performance even under random pruning, supporting the idea of their inherent pruning resistance.

In essence, the ALS method remains effective across all model sizes, offering crucial stability and performance benefits, especially at higher sparsity levels.

Q3. Omission of Llama-3-80B in experiments

Our response

We have conducted comprehensive experiments on the Llama3 70B model. Our findings, detailed in Tables A, B and pdf in the global rebuttal, offer valuable insights into our method's performance on SOTA large language models.

Overall Trends:

(1). Our ALS method consistently outperforms other pruning methods (Magnitude, SparseGPT, and Wanda) across all sparsity levels (50%, 60%, and 70%).

(2). The improvements are particularly pronounced at higher sparsity levels.

Significant Improvements:

(1). Extreme Perplexity Reduction at 60% Sparsity:

• Magnitude pruning: PPL reduces by 341.16 (a remarkable 93.5% reduction)

(2). Consistent Improvements for Top Performers:

• At 50% sparsity, SparseGPT's PPL reduces by 1.25 (14.7% reduction)
• At 70% sparsity, Wanda's PPL reduces by 32.98 (21.2% reduction)

(3). Robustness at 70% Sparsity:

• Magnitude pruning: PPL reduced from 95886.56 to 35844.32 (62.6% improvement). ALS significantly improves performance in challenging scenarios.

In conclusion, our ALS method demonstrates consistent improvements across various sparsity levels and particularly excels at higher sparsities, maintaining reasonable performance even at 70% sparsity.

Reference:

[1] Li, Z., et al. (2020). Train big, then compress: Rethinking model size for efficient training and inference of transformers. ICML 2020.

[2] Liu, S., et al. (2022). The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv:2202.02643.

Dear Reviewer 8iME,

Thank you so much for the detailed and constructive comments. Please see our responses below to your concerns and questions one by one.

Q1. Improving the paper's presentation.

Our Response:

We appreciate the reviewer's suggestions about writing. Based on these recommendations, we commit to carefully revising each paragraph. As part of this revision, the paper's clarity will improve through several changes as follows. Long sections will be divided into concise paragraphs, each beginning with an informative topic sentence. To guide readers, we'll introduce strategic subheadings such as normalization techniques and reweighting strategies. Furthermore, we'll create smooth transitions between paragraphs and sections, ensuring a logical flow of ideas throughout the paper.

Q2. Evaluation of inference efficiency.

Our Response:

We've conducted experiments to address concerns. Table E in the global rebuttal showcases results for the LLaMA-V2 7B model across various sparsity levels:

Results show that 50% sparsity achieves a 2.355x speedup, reducing latency from 58.1968 ms to 24.7099 ms.

These results highlight our method's ability to maintain model performance while delivering substantial computational gains.

Notably, at 50% sparsity, we observe a significant reduction in memory usage, as the dense model uses 27.6 GB, while the 50% sparse model uses only 13.8 GB, achieving a 50% reduction. For full details, please refer to Table F in the global rebuttal.

Q3&Q4. Computation efficiency compared to existing baselines and overhead of solving the linear programming problem

Our Response:

Our method's computational efficiency is demonstrated through experiments on the LLaMA V2 7B model. The total cost of our approach includes two main components: Redundancy Metric (RM) calculation (90 seconds) and Linear Programming (LP) solution (160 milliseconds). For Wanda/SparseGPT/Magnitude w ALS, these times are added to their respective pruning times.

In comparison, BESA [50] requires 4.5 hours for sparsity allocation and pruning, while our approach is significantly faster, completing the process in minutes rather than hours.

For acceleration, we optimize calculations for large models (81 blocks). We use an 80 x 80 RM matrix plus 7 additional 7 x 7 matrices, instead of a large 7 (components within a block) x 80 (blocks) x 7 x 80 matrix.

Q5. Performance improvement decrease for larger models

Our response :

The apparent decrease in performance improvement for larger models is an interesting trend we’ve observed with our ALS method. This aligns with broader patterns in model pruning and compression, where diminishing returns are common as models grow larger. However, despite this general trend, our ALS method demonstrates consistent performance improvements across various model sizes, with particularly notable effects on larger models and at higher sparsity levels:

(1). Significance of Relative Gains

While absolute improvements may appear smaller for larger models, the relative gains remain substantial. For Llama3 70B at 50% sparsity, our ALS method improves perplexity from 8.49 to 7.24 with SparseGPT, representing a 14.72% relative improvement. In the context of large language models, such gains can translate to meaningful performance enhancements in real-world applications.

(2). Inherent Capabilities of Larger Models

Larger models exhibit a natural resistance to pruning. As shown in Table 2 of the paper, the LLaMA-V1 65B model at 50% sparsity only increases in perplexity from 4.93 to 7.37 with Wanda, leaving small room for improvement. Nevertheless, our method further reduces this to 7.15. This observation aligns with the findings of Li et al. [1].

(3). Effectiveness at Higher Sparsity Levels

Our method demonstrates particular strength at higher sparsity levels, especially for the Llama3 70B model, as shown in Tables A and B in the global rebuttal. For instance:

• At 70% sparsity, Wanda w. ALS reduces PPL by 32.98 compared to Wanda alone.

• At 60% sparsity, Magnitude w. ALS reduces PPL by 341.16 compared to Magnitude alone.

These significant improvements in performance retention at high sparsity levels are crucial for practical deployments where substantial model size reduction is required.

(4). Alignment with Theoretical Insights

The behavior we observe aligns with recent theoretical insights. Liu et al. [2] demonstrated that larger models can maintain performance even under random pruning, supporting the idea of their inherent pruning resistance.

In essence, the ALS method remains effective across all model sizes, offering crucial stability and performance benefits, especially at higher sparsity levels.

Reference:

[1] Li, Z., et al. (2020). Train big, then compress: Rethinking model size for efficient training and inference of transformers. ICML 2020.

[2] Liu, S., et al. (2022). The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv:2202.02643.