MoDeGPT: Modular Decomposition for Large Language Model Compression

Q2: Rank of matrices in the experiments

Throughout our experiments, we selected ε values that maximize sparsity levels around 20–30%. This approach avoids extreme sparsity while maintaining a certain level of heterogeneity, which proved to be very effective when compared against other state-of-the-art global allocation strategies, as shown in Appendix B.9 and the general response.

In Appendix B.9, we present the resultant ranks for 30% compression of the LLaMA-2 7B and 70B models, as shown in Table 26 and Figure 10. These results were obtained using our global sparsity allocation strategy with ε = 0.1 and ε = 0.02 for the 7B and 70B models, respectively.

A general trend observed across different model sizes includes:

• Ranks peak in the very first and last layers.
• Ranks are minimal across approximately 75% of the model's depth.
• Rank distributions are remarkably similar for both models, suggesting a deep connection between the allocation strategy and the LLaMA-2 architecture.

As a demonstration, the table below shows the ranks of the key, query, and value projection matrices of the Llama-2 7B model in every layer, the ranks for 70B model can be found in Table 26 in Appendix B.9:

---

References

[1] Ashkboos, S., et al. "SliceGPT: Efficient fine-tuning of large language models by slicing and pruning," 2024.
[2] Men, H., et al. "ShortGPT: Compressed language models for faster inference and reduced memory footprint," 2024.
[3] Yin, X., et al. "Outlier-aware layer sparsification for efficient neural networks," 2023.

Dear Reviewer dBU828, we appreciate your time and insightful feedback. We especially thank you for evaluating our work as "novel" and recognizing the literature review and analysis as "comprehensive". We greatly appreciate your positive comments!

---

W1 & Q1: Intrinsic bias and model generalizability on more diverse zero-shot tasks

We evaluated our model’s generalizability on additional zero-shot tasks, including OpenBookQA, COPA, Lambada, MMLU, and BoolQ, and compared it against decomposition and layer-pruning baselines SliceGPT [1] and ShortGPT [2].

The comparisons were conducted on 30% compressed Llama-2 7B. From the table below, we observed that MoDeGPT consistently outperforms the baselines across a diverse range of tasks**, demonstrating its robustness and generalizability.

Additionally, the relative performance degradation compared to the dense model on these tasks:

• The degradation is most significant for Lambada, with around 16.7% drops compared to the average 7.2% drop.
• However, similar task biases are observed for the baselines, with over 40% degradation on the task. This suggests that Lambada is intrinsically sensitive to model compression.

Despite this sensitivity, MoDeGPT exhibits significantly better resistance to performance degradation, with reductions of only 33% to 50% of the baseline degradation levels. This highlights our method’s advantage on tasks sensitive to compression.

Notably, on the COPA task, MoDeGPT achieves zero degradation, suggesting that it is particularly well-suited for this task. Overall, while our method shows some intrinsic bias, it demonstrates strong and consistent performance across diverse tasks and superior robustness on compression-sensitive tasks.

---

---

W2: Overfitting of the model to calibration data

While calibration with a specific dataset may risk overfitting, our new experiments on layer sparsity allocation comparisons revealed that our global sparsity allocation improves resistance to overfitting compared to baselines.

In these experiments, we used MoDeGPT as the base compression method, combined with our global sparsity allocation, the state-of-the-art allocation strategy OWL, and uniform allocation. The following key observations were made:

• While OWL achieves better perplexity, our sparsity allocation outperforms OWL on every downstream task. This indicates that OWL may overfit the calibration data, as its low PPL does not translate to better generalization in downstream tasks.
• Additionally, our method outperforms uniform allocation, demonstrating that global sparsity allocation not only enhances task generalization but also mitigates overfitting compared to the baseline.
• By inspecting the sparsity standard deviation (visualized in Figure 9, Appendix B.9), we observed that our sparsity distribution is more heterogeneous. This suggests that heterogeneity plays a critical role in improving task generalization and preventing overfitting.

---

---

W4: Justification for efficiency

Although MoDeGPT requires a longer compression time compared to methods like SliceGPT and layer-pruning approaches such as ShortGPT and SLEB, it demonstrates superior efficiency in terms of both accuracy and cost. To substantiate MoDeGPT's cost and accuracy efficiency despite its longer compression time, we conducted evaluations under identical computational budgets (accounting for both compression and recovery fine-tuning using LoRA) and compared its accuracy performance against baseline methods.

Specifically, we:

1. Conducted experiments on 30% compressed Llama-2 7B using the Alpaca dataset for calibration.
2. Adjusted the fine-tuning epochs for each method to equalize total computational budgets, accounting for differences in compression times.
3. Fixed the LoRA parameters to be consistent across all methods (lora_alpha=10, lora_r=32).

The table below presents zero-shot accuracies before and after fine-tuning (shown as after/before).

Key Insights:

1. MoDeGPT outperforms baselines:

   • MoDeGPT achieves the highest zero-shot accuracy across all tasks (excluding perplexity), both before and after fine-tuning.
   • Its superior performance is primarily attributed to the compression phase.

2. Importance of compression:

   • The better perplexity but worse zero-shot performance of SliceGPT compared to MoDeGPT highlights the critical importance of the compression phase. Excessive focus on fine-tuning can exacerbate overfitting and underperform compared to a well-compressed model.

3. SLEB's limited gains:

   • Despite its long fine-tuning time, SLEB achieves smaller improvements than SliceGPT in zero-shot performance, further emphasizing the pivotal role of compression in determining final performance.

4. Effectiveness without fine-tuning:

   • MoDeGPT outperforms baselines even without fine-tuning, showcasing its effectiveness during the compression phase.

In conclusion, while MoDeGPT has a longer compression time compared with some other baselines, it achieves the best performance under the same computation budget, which justifies that our method is also cost-efficient.

---

---

W5: Comparisons with layer pruning methods

We include comparisons of perplexity (lower the better) with layer pruning strategies SLEB [3] and ShortGPT [2] for 7B and 13B models below (see the table in the reply to Q4 for 70B comparisons), showing that MoDeGPT outperforms in all cases.

Additionally, Figure 3 in the main text demonstrates that MoDeGPT achieves a superior trade-off between perplexity and throughput.

---

Dear reviewer XvJF31, we appreciate your time and thoughtful feedback. Especially, thank you for evaluating our work as “novel”, having “exhaustive experiments”, and recognizing our writing as “well-organized”. We are sincerely encouraged by your thoughtful comments!

---

W1: SliceGPT's adapter overhead can be eliminated

While removing the adapters does not lead to dimensional errors, it significantly reduces performance. To substantiate this, we evaluated the zero-shot performance of SliceGPT with and without adapters, as detailed in the table below. The experiments were conducted using Llama-2 7B with 30% compression, fine-tuned on the Alpaca dataset.

The results demonstrate that the adapters are indispensable for maintaining model performance.

---

---

W2 & Q5: Characterizations of module and justifications for the decompositions

We have revised Section 3.2 to provide clearer intuition and justification for our approach. Below is a brief summary of the key rationale:

• The main reason for selecting different decompositions is the number of nonlinear functions in each module, which varies across modules:

  • Type-1 module: 1 nonlinear function.
  • Type-2 module: 2 nonlinear functions.
  • Type-3 module: 0 nonlinear functions.

• This variation leads to differing levels of complexity when solving the proposed modular decomposition problem (Equation 6). Specifically:

  • For matrices embedded within nonlinear functions, directly solving Equation 6 without any structural constraints on the compressed matrix is intractable.
  • To address this, we constrain the compressed matrix to the form of a multiplication by a column selection matrix (Section 3.2).

• However, this constraint introduces additional challenges:

  • The column selection matrix is highly structured, with only one non-zero element per column.
  • Consequently, standard methods such as SVD are not suitable, as they generally produce dense matrices, which conflict with the desired structure.

• Our technical contribution lies in deriving closed-form optimization solutions for these cases:

• Depending on the number of matrices involved in column selection matrix multiplication, the optimal solutions correspond to different decompositions:

  • Type-3 (0 nonlinear functions):
    The compressed matrices are without constraints, so we can simply use SVD to obtain the optimal solution.
  • Type-1 (1 nonlinear function):
    Only one matrix is multiplied by a column selection matrix, and solving this selection matrix is equivalent to finding the optimal landmarks, as in the Nystrom approximation.
  • Type-2 (2 nonlinear functions):
    Two matrices are multiplied by a shared column selection matrix. As the key and query are multiplied together, the selection matrix can be solved by finding the optimal landmarks as in the CR decomposition. These connections are formalized in Theorems 1, 2, and 3.

---

W3 & Q2: lack of details in the proof of Theorem 4 and the assumption is too strong

We have revised the proof to include all the necessary details, as provided in Appedix A.4.

Regarding $\varepsilon$, we would like to emphasize that we do not assume it to be infinite. Instead, we take the limit simply to demonstrate the existence of a sufficiently large number $N$, such that when $\varepsilon > N$, the proposed solution in Equation 11 is optimal.

This explanation has been addressed with mathematical rigor in the updated Appendix A.4.

---

Q4: Does MoDeGPT outperforms competitors in large models (70B)?

• We conducted new experiments on the Llama2-70B model, yielding even more promising results than smaller models. Notably, we achieved:
  • 4.5% and 3.2% drops in performance with 30% compression, using only 128 calibration samples from WikiText-2 and Alpaca, respectively, without recovery fine-tuning.
  • These results outperform decomposition and layer pruning baselines, including SliceGPT [1], ShortGPT [2], and SLEB [3].

---

Q5: What are the characteristics of the proposed method, and why to use them?

Please refer to the response to W2 above.

---

References:
[1] Ashkboos, S., et al. "SliceGPT: Efficient fine-tuning of large language models by slicing and pruning," 2024.
[2] Men, H., et al. "ShortGPT: Compressed language models for faster inference and reduced memory footprint," 2024.
[3] Song, Y., et al. "SLEB: Structured Layer-wise Efficient BERT Pruning for Large-Scale Pre-trained Models," 2024.
[4] Yin, X., et al. "Outlier-aware layer sparsification for efficient neural networks," 2023.
[5] Yuan, X., et al. "LLM-Pruner: On the Structural Pruning of Large Language Models," arXiv preprint arXiv:2305.11627, 2023.

W6: Unclear presentation of challenges in introduction.

We have revised the second paragraph of the introduction to clearly summarize the challenges by adding the lines "In summary, matrix decomposition approaches either (i) discard a large portion of ranks, or (ii) introduce substantial parameter overheads. These challenges significantly hinder the effective reduction of parameters without compromising accuracy."

---

W7: Ambiguous criteria in Table 1

1. Backward Propagation and Memory Efficiency:
   Although not relying on backward propagation does not necessarily result in a faster algorithm, it is usually more memory efficient. Backward propagation often consumes many times the memory of the model size, making its avoidance desirable for limited-resource environments. For instance, our algorithm can run on a single GPU for a 13B model, whereas methods relying on backward propagation, such as LLM Pruner [5], require at least two GPUs and consume over 100GB of memory in our experiments.

2. Maintaining accuracy without fine-tuning:
   We agree that this criterion could lead to ambiguities. Based on your feedback, we have removed this criterion from Table 1. Thank you for the suggestion.

3. Structured vs. semi-structured:
   We have revised Table 1 to better emphasize fully-structured methods and explicitly denote SparseGPT as semi-structured. We believe this distinction is significant, as semi-structured methods require special GPU support to achieve real-time speedup, creating a gap in practical applicability.

---

Q1: Can MoDeGPT outperforms others with the same pruning cost?

Please refer to the response to W4 above.

---

Q2: More details on the proof of Theorem 4.

Please refer to the response to W3 above.

---

Q3: Does the proposed Global Sparsity Allocation outperform OWL [4]'s strategy?

We updated Appendix B.9 to include experiments comparing our method with the state-of-the-art allocation approach OWL [4] and uniform allocation** as baselines on 30% Llama2-7B compression in the first table below (Table 27 in Main).

In these experiments, we used MoDeGPT as the base compression method combined with our global sparsity allocation, OWL, and uniform allocation. Key observations from the results are as follows:

• While OWL achieves better perplexity, our sparsity allocation outperforms OWL on every downstream task. This suggests that our method might be more generalizable. -By inspecting the sparsity standard deviation (a visualization of the distribution difference is also provided in Figure 9 in Appendix B.9), we found that our distribution is more heterogeneous. This observation suggests that heterogeneity could play an important role in enhancing task generalizability.
• The results are consistent with the findings from the Llama2-70B experiments, as shown in the second table below (Table 6 in Main).

---

---

---

Q3: Discussion of different compression rates among modules

While our work uses nonuniform sparsity across layers, we apply uniform sparsity across modules within the same layer. To investigate the impact of heterogeneous sparsity across modules, we conducted additional experiments by varying the compression rates for MLP and attention blocks while keeping the overall compression rate fixed at 30%. The results are presented in the table below and have also been updated in Appendix B.10.

We found that applying larger compression rates to attention blocks improves perplexity but leads to poorer zero-shot task performance. This observation suggests that such a distribution might be more prone to overfitting during calibration.

The table also demonstrates that performance is sensitive to unequal sparsity distribution across modules, indicating that a more sophisticated allocation strategy might be necessary to outperform the uniform strategy.

---

Table: Heterogeneous Sparsity Allocation in Modules

---

Q4: Why is MoDeGPT more efficient than the baseline at the same compression rate (Figure 3)?

MoDeGPT achieves superior efficiency in the accuracy and throughput trade-off, as shown in Figure 3, where its line lies in the bottom-right corner. This is due to the following factors:

1. Mathematically Principled Compression

   • MoDeGPT’s compression method is principled and mathematically grounded, with all compressions derived using closed-form expressions.

2. Fully-Structured Compression for Speedup

   • MoDeGPT leverages fully-structured compression, resulting in descent throughput speedup without the need for specialized GPU support. In contrast, methods like ShortGPT and SLEB rely on coarse compression strategies (e.g., layer pruning), achieving faster speedups but at the cost of accuracy loss.

3. Modular Output Optimization

   • Unlike decomposition-based approaches such as SliceGPT or SVD, which optimize individual matrix independently, MoDeGPT minimizes the modular outputs by jointly optimizing a pair of matrices. This approach better aligns with the global behavior of the LLM’s output, ensuring superior downstream performance. However, this also introduces greater algorithmic challenges, which MoDeGPT successfully addresses.

---

Dear Reviewer ZuW603, Thank you for your time and thoughtful feedback. We particularly appreciate your recognition of the GQA-modified algorithm presented in the appendix and your positive comments. Below, we address your concern regarding the justification of different decompositions in our approach

---

W1: Justification for choice of decomposition for different modules

We have revised Section 3.2 to provide clearer intuition and justification for our approach. Below is a brief summary of the key rationale:

• The primary reason for selecting different decompositions lies in the number of nonlinear functions present in each module, which varies across the modules:

  • Type-1 module contains 1 nonlinear function.
  • Type-2 module contains 2 nonlinear functions.
  • Type-3 module contains 0 nonlinear functions.

• This variation leads to differing levels of complexity when solving the proposed modular decomposition problem (Equation 6). Specifically:

  • For matrices embedded within nonlinear functions, directly solving Equation 6 without any structural constraints on the compressed matrix is intractable.
  • To address this, we constrain the compressed matrix to the form of a multiplication by a column selection matrix (Section 3.2).

• However, this constraint introduces additional challenges:

  • The column selection matrix is highly structured, with only one non-zero element per column.
  • Consequently, standard methods such as SVD are not suitable, as they generally produce dense matrices, which conflict with the desired structure.

• Our technical contribution lies in deriving closed-form optimization solutions by reformulating the problem into a form solvable using existing matrix decomposition techniques. Depending on the number of nonlinear functions in the module, the optimal solutions correspond to different decompositions:

  • Type-3 (0 nonlinear functions):
    The compressed matrices are without constraints, so we can simply use SVD to obtain the optimal solution.
  • Type-1 (1 nonlinear function):
    Only one matrix is multiplied by a column selection matrix, and solving this selection matrix is equivalent to finding the optimal landmarks, as in the Nystrom approximation.
  • Type-2 (2 nonlinear functions):
    Two matrices are multiplied by a shared column selection matrix. As the key and query are multiplied together, the selection matrix can be solved by finding the optimal landmarks as in the CR decomposition.

• These connections and solutions are formalized in Theorems 1, 2, and 3.

---

Q1: Why does SparseGPT have a fixed compression rate?

SparseGPT employs a semi-structured pruning approach that enforces a specific sparsity pattern known as 2:4 sparsity, where exactly two out of every four elements are set to zero. Due to this special pattern, the compression rate is strictly fixed at 50%. Additionally, unlike the fully-structured compression used in our method, this special sparsity pattern requires NVIDIA GPU support for effective real-time acceleration.

---

Q2: Why is the SVD-based method worse than uniform pruning?

Although neither method accounts for input statistics, uniform pruning has a milder impact on the output scale. For instance, with 30% compression, uniform pruning typically reduces the output scale by 30% on average across the entire model.

In contrast, the SVD-based approach can inadvertently remove parts of the input space corresponding to the largest eigenvalues. This leads to a disproportionate reduction in the output scale, making it more variable and less predictable. Such sensitivity in output scaling reduces the stability of the LLM's overall outputs, ultimately degrading downstream performance.

Furthermore, as shown in Figure 1, SVD produces two matrices during compression. Consequently, for a fixed compression rate, the rank reduction is effectively doubled, which further deteriorates the model’s accuracy.

To summarize, two main factors contribute to the inferior performance of vanilla SVD compared to uniform pruning:

1. Instability from input space disruption: While uniform pruning impacts the model evenly, the SVD-based approach introduces instability by disproportionately altering critical components of the input-output relationship.
2. Double rank reduction: The rank is reduced twice during compression to accommodate the two matrices produced by SVD, further compromising the model’s accuracy.

---

Dear Reviewer ZuW6,

Thank you for your thoughtful feedback and for taking the time to review our paper. We hope you had a wonderful Thanksgiving holiday. Inspired by your invaluable suggestions, we have continued refining our methods, and we are excited to share the latest improvements to our methodology, particularly in inference speed.

---

Improved Speed and Accuracy with Nonuniform Module Sparsity

Following your insights and drawing inspiration from prior strategies [3], we refined our global sparsity allocation strategy by introducing distinct sparsity levels for the MLP and MHA blocks within each transformer layer. Instead of calculating a single score per layer, we now compute two scores—one for MLP and one for MHA using the same correlation as described in Section 3.3. The updated global sparsity allocation in Equation 10 is as follows:

$$\max_{\phi_{1:L}}\sum_{i=1}^L\sum_{j \in \{\text{mlp}, \text{mha}\}} w_j (s^j_i (1-\phi^j_i) + \varepsilon H(\phi_i)) \quad \text{such that} \quad \frac{1}{L(w_{\text{mlp}} + w_{\text{mha}})} \sum_{i=1}^L \sum_{j \in \{\text{mlp}, \text{mha}\}} w_j \phi^j_i = \phi_{\text{avg}}, \quad 0 \leq \phi_i \leq 1,$$

where $\phi^j_i$ and $s^j_i$ represent the sparsity and score for the $j$-th block in layer $i$, respectively, and the weights $w_{\text{mlp}}=2, w_{\text{mha}}=1$ are applied to preserve the average sparsity, consistent with the parameter size ratio in transformer blocks. The solution has a similar closed-form solution:

$$\phi = L(w_{\text{mlp}} + w_{\text{mha}})\phi_{\text{avg}}\times\text{Softmax}(-s\odot w/\varepsilon).$$

This updated strategy has enhanced both compression accuracy and inference throughput, especially in the inference speed. Notably, in our 30% compression experiments on Llama2-7B (as shown in the table below), we achieved the fastest throughput among all baselines (even faster than layer pruning strategies!) while maintaining superior accuracy. Our updated allocation rule is consistent with your insight that our method can benefit from higher sparsity in MHA.

Importantly, these updates come with minimal computational overhead. Although we now calculate two scores per layer (instead of one), the computational cost is negligible as score calculation remains lightweight and does not increase compression time.

We are thrilled to share these findings and will include comprehensive experiments in the revised paper. Thank you for your insightful feedback and the time spent during the rebuttal period, which have greatly enhanced this research.

---

References

[1] Song, Y., et al. "SLEB: Sparsity-aware learning for efficient bandwidth in language models," 2024.
[2] Ashkboos, S., et al. "SliceGPT: Efficient fine-tuning of large language models by slicing and pruning," 2024.
[3] Zhang, X., et al. "FINERCUT: Finer-grained Interpretable Layer Pruning for Large Language Models," 2024.

W2: Lack of Analysis on Global Sparsity Allocation

We updated Appendix B.9 to include experiments comparing our method with the state-of-the-art allocation approach OWL [1] and uniform allocation as baselines on 30% Llama2-7B compression in the first table below.

In these experiments, we used MoDeGPT as the base compression method combined with our global sparsity allocation, OWL, and uniform allocation. Key observations from the results are as follows:

• While OWL achieves better perplexity, our sparsity allocation outperforms OWL on every downstream task. This suggests that our method might be more generalizable. -By inspecting the sparsity standard deviation (a visualization of the distribution difference is also provided in Figure 9 in Appendix B.9), we found that our distribution is more heterogeneous. This observation suggests that heterogeneity could play an important role in enhancing task generalizability.
• The results are consistent with the findings from the Llama2-70B experiments, as shown in the second table.

---

---

---

Q1: In Table 3, is the main claim that although semi-structured methods may outperform MoDeGPT, they are held back by custom GPU support which hinders research velocity?

Indeed, the main claim is that while semi-structured methods may achieve better performance in specific scenarios, their reliance on custom GPU support significantly limits their practicality and adaptability. For instance, mobile device chips typically lack the necessary hardware support, making these methods inefficient or infeasible in such environments.

---

Q2: The Plot of Perplexity Versus Throughput

We have improved Figure 3 in the main text to better illustrate the trade-off between perplexity and throughput. The relative model sizes are now annotated in the figure for added clarity.
As shown, MoDeGPT achieves the best perplexity-throughput trade-off, with its line positioned in the bottom-right corner of the plot. This demonstrates the effectiveness of our method compared to other approaches.

---

References

[1] Yin, et al. "Outlier-aware layer sparsification for efficient neural networks," 2023.

Dear reviewer Z3HT03, we appreciate your time and insightful feedback. Especially, thank you for evaluating our work as “novel”, “well-written” and for acknowledging that the "results are strong." We greatly appreciate your positive comments! Please see below of our response to your concerns.

---

W1: Justification for choice of decomposition for different modules

We have revised Section 3.2 to provide clearer intuition and justification for our approach. Below is a brief summary of the key rationale:

• The main reason for selecting different decompositions is the number of nonlinear functions in each module, which varies across modules:

  • Type-1 module: 1 nonlinear function.
  • Type-2 module: 2 nonlinear functions.
  • Type-3 module: 0 nonlinear functions.

• This variation leads to differing levels of complexity when solving the proposed modular decomposition problem (Equation 6). Specifically:

  • For matrices embedded within nonlinear functions, directly solving Equation 6 without any structural constraints on the compressed matrix is intractable.
  • To address this, we constrain the compressed matrix to the form of a multiplication by a column selection matrix (Section 3.2).

• However, this constraint introduces additional challenges:

  • The column selection matrix is highly structured, with only one non-zero element per column.
  • Consequently, standard methods such as SVD are not suitable, as they generally produce dense matrices, which conflict with the desired structure.

• Our technical contribution lies in deriving closed-form optimization solutions for these cases:

• Depending on the number of matrices involved in column selection matrix multiplication, the optimal solutions correspond to different decompositions:

  • Type-3 (0 nonlinear functions):
    The compressed matrices are without constraints, so we can simply use SVD to obtain the optimal solution.
  • Type-1 (1 nonlinear function):
    Only one matrix is multiplied by a column selection matrix, and solving this selection matrix is equivalent to finding the optimal landmarks, as in the Nystrom approximation.
  • Type-2 (2 nonlinear functions):
    Two matrices are multiplied by a shared column selection matrix. As the key and query are multiplied together, the selection matrix can be solved by finding the optimal landmarks as in the CR decomposition. These connections are formalized in Theorems 1, 2, and 3.

---

Analysis on Global Allocation Strategy

• We compared our global sparsity allocation strategy with the state-of-the-art allocation method (OWL [4]) and uniform allocation for 30% compression using MoDeGPT:

  • While OWL improves upon uniform allocation, our method demonstrates superior zero-shot task performance for both small (7B, table below) and large (70B, table above) models.

  Method	Sparsity Mean	Sparsity Std	Perplexity ↓	PIQA ↑	HellaS. ↑	WinoG. ↑	ARC-E ↑	ARC-C ↑	Average ↑
  Uniform Allocation	30%	0%	9.06	65.18	55.31	63.69	52.36	30.80	53.47
  Global Sparsity Allocation (Ours)	30%	26.72%	7.51	71.40	63.26	67.32	63.26	38.73	60.79
  OWL [4]	30%	4.46%	6.9	68.17	59.12	65.67	56.9	33.36	56.64

---

References

[1] Ashkboos, S., et al. "SliceGPT: Efficient fine-tuning of large language models by slicing and pruning," 2024.
[2] Men, H., et al. "ShortGPT: Compressed language models for faster inference and reduced memory footprint," 2024.
[3] Song, Y., et al. "SLEB: Sparsity-aware learning for efficient bandwidth in language models," 2024.
[4] Yin, et al. "Outlier-aware layer sparsification for efficient neural networks," 2023.