FASP: Fast and Accurate Structured Pruning of Large Language Models

Novelty

1. The main weakness of this paper lies in its novelty. This paper proposes (1) formulation, (2) importance metric, and (3) restoration method, but all of these ideas can be found in previous works [1, 2, 3, 4] with slight modifications in some cases. In detail, the pruning of neurons is used in [1,2,3] and the importance metric of FASP is a straightforward modification of Wanda [4]; it just sums up the importance score of weights in each column to measure the importance of the column. The restoration method of FASP which solves the least-square problem is also used in previous works [1,2]. Therefore, FASP has limited novelty and originality.

Experiments

2. First of all, what is the definition of the problem that FASP tries to solve? Is the compression constraint related to latency or model size? According to the definition of your problem, you need to add more competitors to verify the performance of FASP; you need to compare FASP with layer pruning algorithms, e.g., SLEB [5] if your problem includes latency constraints or 2:4 pruning algorithms, e.g. Wanda [4], if your problem includes model size in your problem. In other words, there are lack of justifications for selecting baseline methods.

3. The inference speed of the pruned model is quite important since the acceleration after pruning is the major reason why we use structured pruning rather than unstructured pruning. However, there are lack of experiments about the inference speed of the pruned models. The only experimental results can be found in four lines in Section 4.1, but there is no comparison with baseline methods. There is no detailed explanation of the experimental settings, e.g., the lengths of input and output sequences.

4. The authors use old-fashioned models such as OPT and Llama. Considering that Llama3 8B shows more than 30%p higher accuracy than Llama2 7B on ARC-challenge benchmark, recent models might show different patterns to the old-fashioned models used in this paper.

Writing

5. Figures and tables are not arranged properly. For example, Tables 5 and 6 can be placed in a single line. Tables 1 and 2 are also can be placed in a single line if you control column widths.

6. Font sizes in Figures 3 and 4 are too small and hard to read.

7. According to the guidelines of ICLR, table captions must be located on the upper side of the tables.

8. There are lack of explanation about Figure 2 in its caption. What's the meaning of each color?

9. The authors use Section 5 for discussion, but there are only 5 lines of text in this section and lack of meaningful content.

10. The authors use imprecise mathematical notations, such as n(1-s) in line 254; there is no guarantee that n(1-s) is an integer and it requires a floor or ceiling function to become an integer.

11. The name of the proposed method does not contain any characteristics of the proposed method and it is too general. According to the reported experimental results, the method is not significantly "fast" and not significantly "accurate"

12. There is no reproducibility statement which is encouraged by the conference guideline (https://iclr.cc/Conferences/2025/AuthorGuide).

[1] Park, Seungcheol, Hojun Choi, and U. Kang. "Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models." The Twelfth International Conference on Learning Representations. 2024.

[2] Kwon, Woosuk, et al. "A fast post-training pruning framework for transformers." Advances in Neural Information Processing Systems 35 (2022): 24101-24116.

[3] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720.

[4] Sun, Mingjie, et al. "A simple and effective pruning approach for large language models." arXiv preprint arXiv:2306.11695 (2023).

[5] Song, Jiwon, et al. "SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks." arXiv preprint arXiv:2402.09025 (2024).

1. The proposed pruning structure does not seem to be universal and needs to be specifically designed for different models (such as OPT and LLaMA mentioned in this paper).
2. Figures 1 and 2 are oversimplified to summarize the characteristics of the proposed method.
3. In terms of Tables 1,3 and 4, the proposed FSAP has only a slight improvement in pruning time and accuracy compared to FLAP.
4. It is inappropriate to use the same title for Tables 3, 5, and 6, which have different purposes.
5. Only comparative and ablation experiments were covered, lacking some intuitive analytical experiments to explore the intrinsic mechanisms of the proposed FSAP.
6. More experiments or explanations are needed to analyze why the proposed FSAP should not remove the columns of WQ and WK.
7. The authors did not provide any code for review, so the reproducibility of this paper is open to question.
8. The prune metric is similar to Wanda.

1. More perplexity results, like on C4. The comparison of perplexity on WikiText might not be fair. FASP restores the pruned weights on 128 samples from WikiText, making it adapt to WikiText. It's an unfair comparison for methods that don't have this restoration step. I also observe a slightly overfit in Table 2, where FASP's with 10% sparsity can achives a smaller perplexity than the original LLM, i.e. 12.42 vs 12.47. Therefore, I encourage to include more perplexity results, like on C4 (but also use WikiText samples to restore the pruned weights).

2. Lack of recent LLM series. The LLMs used in this paper is OPT and LLaMA (first version), it's better to include some recent series, like Llama-3. It's known that Llama-3 is difficult for quantization, it might also pose challenge to pruning. I understand that most previous works report results on LLaMA and OPT, so you prefer to do experiments on them. But the results on a recent series is also important. You can only compare FASP to the strongest baseline on the recent series.

3. Lack of novelty. From the summary, The innovation from this work seems limited. The adapted Wanda is very similar to Wanda, only adding one summation operation along the column. And the knowledge restoration is basically layer-wise knowledge distillation.

1. Lack of Memory Savings Analysis The paper primarily reports on pruning speed and latency improvements, omitting a detailed analysis of memory savings. Given that structured pruning is often motivated by reductions in memory footprint, a quantitative comparison with other pruning methods on memory consumption would strengthen the paper’s practical impact claims. Including this metric would provide additional insight into FASP’s viability for memory-constrained deployments.

2. Missed Opportunity in Attention Mechanism Pruning The approach skips pruning for 𝑊𝑄 and 𝑊𝐾 layers in self-attention due to observed performance degradation. While pragmatic, this choice leaves redundancy in these attention components. Exploring alternatives like partial pruning of heads or adaptive sparsity allocation within attention could increase FASP’s efficiency while addressing the performance concerns, allowing for a more comprehensive pruning strategy.

3. Limited Hardware Evaluation Scope Benchmarking on edge devices and CPU inference frameworks should have been focused. For a pruning method like FASP, which aims to make large models more accessible on constrained hardware, a broader evaluation on diverse hardware setups. Alternatively, a discussion on anticipated hardware bottlenecks or expected variations in performance would help users assess FASP’s versatility across deployment scenarios.

• While FASP offers practical improvements, its core ideas rely heavily on existing pruning strategies, such as those proposed by Wanda and similar structured pruning frameworks. The novelty primarily lies in the integration of these techniques, which does not constitute sufficient methodological novelty.
• Some recent related work is missing. For example, DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models.
• The experiment did not include the comparison with many existing work, such as the above DISP-LLM paper, Pruner-Zero, etc.