FISTAPruner: Layer-wise Post-training Pruning for Large Language Models

Response to Questions

• Response to Question 1

  Thank you for your question.

  • First, we would like to clarify that SGD with or without momentum cannot directly solve the $\ell_1$-regularized training problem. This is because the $\ell_1$-norm is non-smooth, and gradient-based methods like SGD are not well-suited for handling non-smooth objective functions. Instead, solving such problems typically requires subgradient methods or proximal gradient methods, such as FISTA, which are specifically designed to handle non-smooth optimization.

  • Additionally, our proposed FISTAPruner is a training-free pruning method, which fundamentally differentiates it from training-based methods. Specifically, FISTAPruner optimizes a layer-wise pruning model with an $\ell_1$-norm regularization term, as described in the paper. Unlike $\ell_1$-regularized training approaches, FISTAPruner does not require backpropagation over the entire network. As we emphasized previously in our response to Weakness 2, this training-free nature makes FISTAPruner significantly more efficient than methods that optimize $\ell_1$-regularized non-convex loss functions through training. Given these fundamental differences, we believe a direct comparison between FISTAPruner and training-based pruning methods would not provide meaningful insights.

• Response to Question 2

  Thank you for your question.

  • We chose to evaluate FISTAPruner on OPT and LLaMA-1 models to ensure a fair and consistent comparison with the baseline methods reported in our paper, such as SparseGPT [1], Wanda [2], DSnoT [3], and PERP [4]. These works also evaluated their approaches on OPT and LLaMA-1. By using the same models, we aimed to provide a direct and meaningful comparison of FISTAPruner’s performance relative to these established baselines.
  • To address the applicability of FISTAPruner to more recent models, we also included experiments on state-of-the-art models such as LLaMA-3-8B and LLaMA-3-70B, demonstrating that our method generalizes well to newer models as well.

• Response to Question 4

  Thank you for your question.

  • First, DSNoT and PERP are fundamentally weight adjustment/retraining-based methods aimed at improving efficiency further for an existing pruning method, whereas our approach with FISTAPruner is intentionally designed to achieve optimal results without retraining. This distinction reflects our focus on demonstrating that it is possible to match or even surpass further adjustment/retraining-based methods without the need for additional retraining steps.

  • We have conducted comparisons in several scenarios to highlight this point and have demonstrated that FISTAPruner achieves excellent performance without retraining. Therefore, integrating DSNoT or PERP with FISTAPruner would not align with our objective, as adding a retraining step (even if it improves results) does not serve to further validate our approach or its advantages.

  • Regarding the tables, we believe the current structure effectively communicates our comparisons and findings. Combining the tables as suggested may blur the distinction between retraining-based methods and our retraining-free approach and baseline methods, which is central to our work.

• Response to Question 5

  Thank you for your question. The main point of the "Warm Start" section is to discuss the efficiency and effectiveness of different warm-start strategies within our pruning framework. The key takeaway is that we evaluate how various "warm start" strategies integrate with our FISTAPruner framework, concluding that using Wanda as a warm start is a cost-efficient and effective approach in practice.

[1] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot."

[2] Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models."

[3] Yuxin Zhang, et al. "Dynamic sparse no training: Training-free fine-tuning for sparse LLMs."

[4] Max Zimmer, et al. "Perp: Rethinking the prune-retrain paradigm in the era of LLMs."

• Response to Weakness 2
  • Thank you for your comment. While it is true that our method leverages the existing FISTA algorithm and employs $l_1$-norm regularization, we believe the novelty of our approach lies in the following aspects:
    • No back propagations: A distinguishing feature of our approach is that it eliminates the need for computing the gradient by back propagations, making it highly efficient and particularly suitable for scenarios with limited computational resources. This stands in contrast to typical methods leveraging the $l_1$-norm, which often integrate it as a regularization term into the training loss function to induce sparsity and the pruning process requires expensive back propagations to optimize the loss function [1-2]. We have discussed this type of methods in the Related Work section in lines 109-113.
    • Optimization-driven layer-wise pruning scheme with error correction: Unlike typical post-training method which rely on heuristic approaches or non-convex constraints, our work is the first to leverage the $l_1$-norm to induce sparsity in layer-wise pruning of LLMs. Besides, we introduce an intra-layer error correction mechanism tailored specifically to the LLM pruning domain. While FISTA is a natural choice for solving our formulated model, we have also designed a hyperparameter tuning algorithm to ensure that it achieves the desired sparsity levels and structures effectively.
    • Practical relevance: Integrating all our innovations, we achieve state-of-the-art performance on comprehensive benchmarks, as demonstrated in our experiments.
    • [1] Thu Dinh, Bao Wang, Andrea Bertozzi, Stanley Osher, and Jack Xin. Sparsity meets robustness: Channel pruning for the feynman-kac formalism principled robust deep neural nets.
    • [2] Xiufeng Xie, Riccardo Gherardi, Zhihong Pan, and Stephen Huang. Hollownerf: Pruning hashgridbased nerfs with trainable collision mitigation.
• Response to Weakness 3 and Question 3
  • Thank you for your feedback. The primary objective of our proposed adaptive hyperparameter tuning algorithm is to efficiently determine the optimal value of $\lambda$ that balances the trade-off between sparsity and reconstruction error. As outlined in Section 3.4, we designed a bisection-based tuning algorithm tailored for this purpose. We have rigorously proved in Theorem 3.3 (we appreciate your observation regarding the numbering issue and will address it in the next revision) that with this algorithm, FISTAPruner is guaranteed to converge to the desired sparsity level.
  • We acknowledge the existence of many hyperparameter tuning algorithms for machine learning, including BOHB [1], which you suggested for comparison. However, our problem setting differs significantly from the one addressed by such methods. Specifically, in our framework, we only need to determine a single scalar parameter, $\lambda \in \mathbb{R}$, for each layer these parameters are independent of one another. This reduces the task to a one-dimensional parameter search problem, which can be effectively and efficiently solved using our bisection-based approach. ln contrast, methods like BOHB are designed to optimize over high-dimensional parameter spaces, where the goal is to identify a good combination of multiple parameters. As such, these methods are not ideally suited to our setting.
  • [1] Falkner, Stefan, Aaron Klein, and Frank Hutter. "BOHB: Robust and efficient hyperparameter optimization at scale."

Writing

• Response to Weakness 4

  Thank you for your feedback. We appreciate your suggestion to provide additional explanation about FISTA to enhance understanding. While we agree that more background on FISTA might benefit some readers, it is a well-established method, and detailing its fundamentals may not be necessary. Instead, we have focused on our primary contribution: the development of a novel model for layer-wise pruning and the efficient integration of FISTA with our proposed intra-layer error correction mechanism and hyperparameter tuning algorithm. For completeness, we have included detailed mathematical derivations and an introduction to the FISTA iterations in Appendix B.

• Response to Weakness 6

  Thank you for your suggestions regarding the writing format and style. We will carefully address all these points in the revised version of our paper.

Experiments

• Response to Weakness 7

  Thank you for your feedback. With respect, we do not agree with your suggestions regarding the comparisons of FISTAPruner and structured pruning methods. FISTAPruner is specifically designed for unstructured pruning and 2:4 semi-structured pruning, which are fundamentally different domains from structured pruning. Comparing our method with algorithms designed for structured pruning would not provide a fair or meaningful evaluation, as the constraints differ significantly between these approaches.

Response to Weaknesses 1 and 5 cont'd

As shown in the results above, we summarize the PPL (lower is better) comparison across different sparsity levels as follows:

5% and 10%: intra- and inter- layer error correction $<$ intra-layer error correction only $<$ no error correction;

20%: intra-layer error correction only $<$ intra- and inter- layer error correction $<$ no error correction;

50%: intra-layer error correction only $<$ no error correction $<$ intra- and inter- layer error correction.

First, the results confirm the effectiveness of our intra-layer error correction mechanism, as it consistently outperforms the no-error-correction approach.

Second, the results confirm the effectiveness of using both intra- and inter-layer error correction at low sparsity levels, as it consistently outperforms the intra-layer error correction alone at 5% and 10% sparsity.

Third, the results show that using both intra- and inter-layer error correction is sensitive to sparsity levels and tends to perform worse at higher sparsity. Specifically, at 20% sparsity, it underperforms compared to intra-layer error correction alone, and at 50% sparsity, it even performs worse than the no-error-correction approach.

To explain why the use of both intra- and inter-layer error correction is sensitive to sparsity levels, we believe this occurs because higher sparsity levels make the pruning task more difficult, leading to greater error accumulation across layers. When both intra- and inter-layer error correction are applied, mitigating the accumulated error from previous layers may dominate the optimization objective in deeper layers, causing the pruning performance of the current layer to suffer.

Mathematically, let $W_k$ and $X_k$ represent the weight matrix and the activation of the $k$-th layer in the original network, respectively. Similarly, let $W_k^*$ and $X_k^*$ denote the pruned weight matrix and the corresponding activation in the pruned network. In a layer-wise pruning scheme with both intra- and inter-layer error correction mechanisms, we minimize the loss for each layer individually:

$X_k$ depends on the activation from the previous layer:

where $f_k$ represents some operations (e.g., activation function or normalization). Therefore, we can express the pruned activations recursively as:

The error at layer $k$ is defined as:

Therefore, under high sparsity level, this amplification often results in the accumulated error $\Delta X_k$ becoming dominant at deeper layers.

Thus, for large $k$, considering both intra- and inter-layer error correction mechanism, there is:

As a result, the optimization process shifts focus towards correcting this accumulated error rather than pruning the current weight matrix $W_k$.

In other words, minimizing the term primarily addresses the error correction from previous layers rather than properly pruning the weight matrix $W_k$, which negatively impacts the pruning performance in deeper layers.

Response to Weaknesses

Method

• Response to Weaknesses 1 and 5

  • Thank you for your comment. While the similar concept of error correction has been explored in the context of CNNs (e.g., [1]). To the best of our knowledge, our approach first specifically tailors these principles to the unique architecture and requirements of LLMs. Existing work in LLM pruning, such as SparseGPT [2] and Wanda [3], do not include error corrections at all. To implement intra-layer error corrections, it is necessary to pass the original activation ($X$) to the original weight ($W$), while simultaneously passing the activation from the previously pruned operator ($X^*$) to the target pruned weight ($W^*$). This results in the model $||W^* X^* - WX||_F$. Based on the code released by SparseGPT and Wanda, it is evident that they utilize the same activation within a decoder layer: $||W^*X - WX||_F$. Moreover, they forward the activation from one pruned decoder layer to the subsequent layer, complicating their pruning framework and creating dependencies across decoder layers.

  • [1] He, Yihui, Xiangyu Zhang, and Jian Sun. "Channel pruning for accelerating very deep neural networks."

  • [2] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot." International Conference on Machine Learning. PMLR, 2023.

  • [3] Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models."

  • We apply only the intra-layer error correction mechanism for two reasons:

    1. Parallelization: Intra-layer error correction enables independent pruning of each decoder layer, allowing us to distribute the pruning task across multiple devices by assigning different decoder layers to different devices. This will increase the overall pruning efficiency.
    2. Sparsity Sensitivity: While combining intra- and inter-layer error correction could intuitively reduce error accumulation across the network, we found that this approach is effective only at low sparsity levels. When the pruning task becomes harder (i.e., higher sparsity), global error correction tends to overshadow the pruning process of individual layers, ultimately leading to worse performance.

    The first reason is straightforward; we will explain the second reason in more detail below.

    We conducted a series of comparison experiments on OPT-125M at sparsity levels of 5%, 10%, 20%, and 50%. The experiments included three conditions: intra-layer error correction only, both intra- and inter-layer error correction, and no error correction. The results are presented in the following tables.

    OPT-125M under 5% Sparsity	WikiText	C4	PTB
    Intra-layer Error Correction Only	27.64	26.57	38.99
    Intra-layer and Inter-layer Error Correction	27.63	26.56	38.98
    No Error Correction	27.69	26.60	38.98

    OPT-125M under 10% Sparsity	WikiText	C4	PTB
    Intra-layer Error Correction Only	27.47	26.59	39.00
    Intra-layer and Inter-layer Error Correction	27.43	26.58	39.04
    No Error Correction	27.52	26.69	39.07

    OPT-125M under 20% Sparsity	WikiText	C4	PTB
    Intra-layer Error Correction Only	27.36	26.71	39.39
    Intra-layer and Inter-layer Error Correction	27.37	26.72	39.53
    No Error Correction	27.61	26.91	39.85

    OPT-125M under 50% Sparsity	WikiText	C4	PTB
    Intra-layer Error Correction Only	33.54	30.93	49.79
    Intra-layer and Inter-layer Error Correction	35.90	32.93	55.24
    No Error Correction	34.48	32.24	54.11

Response to Weakness 1

Thank you for your insightful feedback and for pointing out the related work that we initially missed. We acknowledge that error correction techniques, including those used in the repositories you mentioned (OmniQuant and K-prune), are indeed relevant to the discussion. We have discussed these works in the related work section (line 130 - 138) in the revised paper.

However, we would like to emphasize that the unique contribution of our approach lies in the application of error correction specifically at the intra-layer level. This targeted method not only enhances accuracy but also facilitates parallelization, as detailed in our initial response to Weaknesses 1 and 5. We believe this distinction sets our work apart from existing methodologies. We have updated our paper to clarify this distinction and to better highlight the advantages of our intra-layer error correction mechanism (see lines 136-138, 180-182, 462-477, 933-1035).

Response to Weakness 2

Thank you for your detailed comments. We appreciate the opportunity to clarify this point.

• Regarding the capability of using SGD to solve non-smooth optimization problems, we agree that SGD can be applied with certain modifications, and our initial response explicitly stated that SGD cannot be directly applied to optimize $\ell_1$-regularized problems. Specifically, in the examples you mentioned (e.g., replacing the gradient at non-differentiable points with constants), these approaches effectively replace the gradient with a subgradient, which aligns with our discussion of subgradient methods in the original response. However, it is important to note that subgradient-based approaches typically exhibit a slower convergence rate of $\mathcal{O}(1/\sqrt{k})$ compared to the $\mathcal{O}(1/k)$ rate achieved in smooth optimization. In contrast, FISTA achieves an accelerated convergence rate of $\mathcal{O}(1/k^2)$, as rigorously proven in [1] and mentioned in Remark 1 of our paper. We have included this analysis in lines 213-215. Furthermore, we carefully examined the works you referenced, and we observe that many of these methods rely on relaxations of the non-smooth regularization terms (e.g., replacing the $\ell_0$-norm with a differentiable approximation or surrogate loss). These relaxations enable the use of SGD on the modified, smoother objective function. As such, it is not that SGD itself is inherently well-suited for non-smooth optimization; rather, it becomes applicable through these relaxations or approximations. In summary, while SGD with modifications can be applied to certain non-smooth problems, these approaches are generally slower and less efficient than FISTA in terms of convergence rate. Moreover, many of the referenced works rely on problem-specific relaxations to enable SGD's use, which differs fundamentally from the proximal optimization framework employed by FISTA.

Dear Reviewer EAKc,

Hi, as the discussion period is set to end soon, we kindly ask if you have any further questions or comments, or if there are any aspects of the paper that require additional clarification. Your feedback is invaluable to us, and we want to ensure we address any remaining concerns.

If there are no further concerns, we would greatly appreciate it if you could kindly update your score accordingly.

Best regards,
Authors 8547

Thank you for your detailed response. However, my major concerns regarding your main ideas remain unresolved.

[Error Correction]

The authors have not conducted sufficient surveys on error correction algorithms in existing works. Specifically, the intra-layer error correction algorithm, which is one of the main ideas of this paper, is not a novel approach. It has already been studied in previous research, as demonstrated in [1].

• [1] El Halabi, Marwa, Suraj Srinivas, and Simon Lacoste-Julien. "Data-efficient structured pruning via submodular optimization." Advances in Neural Information Processing Systems 35 (2022): 36613-36626.

[Use of FISTA]

The efficiency of solving convex problems, such as SparseGPT, lies in its ability to prune large models like Llama-3 70B in approximately two hours. If there is a more moderate time limits, such as 12 hours, block-wise pruning with SGD emerges as a strong competitor. Here, a block refers to the combination of Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP) modules.

FISTAPruner requires nearly 12 hours for pruning, undermining its efficiency advantage. Furthermore, the authors do not compare their approach against a straightforward baseline: block-wise pruning with L1 regularization. Notably, block-wise pruning has the potential for higher accuracy, as it accounts for dependencies between linear operations within a block (see OmniQuant referenced above).

[Hyperparameter Optimization]

While the authors discuss the superiority of their hyperparameter search algorithm over BOHB, they provide no experimental results to substantiate this claim, not even from previously published studies. Moreover, BOHB is just one example among numerous existing hyperparameter search algorithms. The authors fail to justify the necessity of proposing a new hyperparameter optimization algorithm rather than leveraging established methods. Additionally, the paper lacks an analysis of the impact of their hyperparameter optimization algorithm on the overall results.

The authors rely solely on their assertions to respond during the long discussion period, without providing additional experiments or surveys on prior research. Furthermore, their responses include many statements that are difficult to agree with, such as "L1 loss cannot be trained using SGD." Therefore, I believe my concerns have not been adequately addressed, and I will maintain my rejection score.

Thank you for your response.

[Error Correction]

We have already addressed this point in our previous response (Part 1 - Response to Weakness 1). Specifically, we acknowledged that error correction techniques, including those used in repositories such as OmniQuant and K-prune, are relevant to our discussion. These works are discussed in the related work section (lines 130-138) of the revised paper. However, we emphasize that the unique contribution of our approach lies in applying error correction at the intra-layer level, which offers both higher accuracy (compared with intra- and inter-layer error corrections) and parallelization advantages. This distinction is clearly stated in the revised paper (lines 136-138, 180-182, 462-477, 933-1035), as noted in our previous response.

Additionally, you mentioned that intra-layer error corrections are used in [1]. However, this is not accurate. In [1], the authors considered three methods: LayerInChange, SeqInChange, and AsymInChange, whose essential ideas correspond to (1) no error correction, (2) inter-layer error correction only, and (3) both intra- and inter-layer error corrections. Thus, intra-layer error correction was not explored in the way we apply it in our approach. Furthermore, [1] is designed for CNNs, whereas our work focuses on LLMs (transformer-based networks).

[1] El Halabi, Marwa, Suraj Srinivas, and Simon Lacoste-Julien. "Data-efficient structured pruning via submodular optimization." Advances in Neural Information Processing Systems 35 (2022): 36613-36626.

[Use of FISTA]

First, we respectfully disagree with the comment, "The efficiency of solving convex problems, such as SparseGPT, lies in its ability to prune large models like Llama-3 70B in approximately two hours." To clarify, SparseGPT does not solve a convex problem. Specifically, the problem they aim to solve, as outlined in Equation (1) in SparseGPT paper [1], is non-convex because it involves discrete mask variables. Moreover, SparseGPT utilizes the OBS framework, which is heuristic-based and operates in a greedy manner. As such, SparseGPT's efficiency does not stem from solving a convex problem, but rather from its heuristic approach, which yields approximate solutions to the underlying non-convex model.

While the subtitle of this paragraph in your comments mentions the "use of FISTA", it appears that your primary concern is not the adoption of FISTA but rather the suggestion of applying SGD to a block-wise L1-norm regularization problem. To our knowledge, this is a novel hypothesis and has not been explored in previous work. It is important to note that introducing block-wise pruning will make the problem non-convex, which presents additional challenges compared to our layer-wise model, which is convex.

We also want to clarify that we have provided theoretical convergence results for SGD, SGD with subgradients, and FISTA in our previous response (Part 1 Response to Weakness 2) and in the revised paper (lines 204-209), demonstrating that FISTA offers the fastest convergence rate.

Additionally, our method focuses on the efficiency and scalability of layer-wise pruning using FISTA, which benefits from the convexity of the problem and provides strong theoretical guarantees for convergence.

[1] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot." International Conference on Machine Learning. PMLR, 2023.

[Hyperparameter Optimization]

While we acknowledge that BOHB (or other similar works) is a generic method for hyperparameter tuning, we would like to emphasize that it is not always the most efficient or appropriate choice for every problem. In our case, we are dealing with a specific 1-dimensional search problem (tuning the sparsity parameter $\lambda$, where the structure of the problem makes the bisection method far more efficient than BOHB.

In fact, we believe that for each algorithm and problem, it is important to design hyperparameter tuning methods that are tailored to the specific needs and properties of that problem. This approach not only leads to better performance but also ensures computational efficiency. While BOHB may work well for high-dimensional or complex problems, applying it to a simpler problem—like ours—would introduce unnecessary complexity and computational overhead, without offering any practical advantage.

Our choice to use a specialized method for this particular scenario is, in our opinion, the most reasonable and effective way to proceed. We do not believe that it is necessary or productive to apply a generic algorithm like BOHB in every case.

Response to Weakness 1 and Question 1

Thank you for your comment. We acknowledge that both SparseGPT [1] and our work approach pruning as an optimization problem. However, there are critical distinctions that highlight the novelty of our contributions.

First, while SparseGPT formulates the pruning objective as minimizing the Frobenius norm $||W^* X - WX||_F^2$, subject to sparsity constraints

it does not incorporate error corrections into this minimization and the problem is inherently non-convex due to the sparsity constraint $W^* \in \mathcal{C}$.

Moreover, SparseGPT does not directly solve the above model but finds approximate solutions by a greedy procedure: following the OBS framework [2], it iteratively removes entries with the smallest impact on the objective function and updates the remaining weights. This method lacks optimality guarantees and may lead to unstable performance.

In contrast, our work introduces an innovative transformation of the non-convex model into a convex one by adding an $\ell_1$-norm regularization term. We then utilize FISTA to find a global solution. This systematic approach not only enhances stability but also gives better pruning outcomes.

Response to Weakness 2 and Question 2

Thank you for your comment. To understand why FISTAPruner achieves significant improvements in the 2:4 pruning setting, we conducted ablation studies specifically focused on 2:4 semi-structured sparsity.

The key components of FISTAPruner for 2:4 semi-structured pruning are:

1. the mathematical formulation of the problem as an $\ell_1$-norm regularized convex optimization problem where the parameter $\lambda$ that balances the sparsity and the reconstruction error is determined by our proposed adaptive tuning algorithm;
2. the intra-layer error correction mechanism that avoids error accumulation;
3. the hard thresholding step after the FISTA iterations to project onto the 2:4 semi-structured space.

While the first component includes several features, they are interdependent; removing any one of them will fail the system. The hard thresholding step is essential to guarantee that the final output adheres to the 2:4 sparsity structure, so ablation studies on this aspect would not yield meaningful insights. Consequently, the only feasible ablation study focuses on the intra-layer error correction mechanism.

We tested FISTAPruner with and without the proposed correction mechanism on OPT-125M, and found that error corrections are vital to the significant improvements of FISTAPruner. The results are summarized below:

We could see that without intra-layer error corrections, FISTAPruner is still able to produce better results. However, incorporating the intra-layer error correction mechanism leads to a substantial drop in perplexity. Thus, we believe that intra-layer error corrections play an important role in the 2:4 semi-structured pruning. This is probably because 2:4 semi-structured sparsity brings more constraints to the sparsity structure so the error accumulation is also more evident. Thus, incorporating error corrections is able to give FISTAPruner the right direction to optimize and thus yields large improvements.

Response to Weakness 3 Thank you for your feedback. We appreciate your concern regarding the nature of our ablation studies. We included the "Amount of Calibration Data" section to the ablation study part following SparseGPT [1]. The "Warm Start" section aimed to explore the effects of initializing the FISTA iterations at different points. However, we agree with you that these do not constitute proper ablation studies. According to your suggestion, we will move both sections to the discussion part of the paper.

[1] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot." International Conference on Machine Learning. PMLR, 2023.

[2] Hassibi, Babak, David G. Stork, and Gregory J. Wolff. "Optimal brain surgeon and general network pruning." IEEE international conference on neural networks. IEEE, 1993.

Response to Major Question:

Thanks for your question. To clarify, both Wanda and our method aim to minimize the discrepancy of the outputs between the pruned network and the original one under sparsity constraints.

To achieve this goal, Wanda employs a heuristic method to prune weights based on a pruning metric. They choose $|W| \|X\|_2$ as their metric, which intuitively considers both the magnitude of the weights $W$ and the input $X$. This metric is designed based on the insight that the computation in a linear layer is $WX$. Consequently, Wanda heuristically uses $|W| \\|X\\|_2$ to identify and prune weights that may have small impact on $WX$.

However, our work formulates the layer-wise pruning problem as an optimization model and solves it to obtain the pruned weights. To do so, we include $\\|W^* X^* - WX\\|_F$ as part of our objective to minimize the discrepancy between the pruned network and the original model, while using the $\ell_1$-norm to induce sparsity. Additionally, by solving our proposed model, we not only prune weights but also update the remaining weights to further minimize $\\|W^* X^* - WX\\|_F$.

In summary, here are the similarities and differences between the objectives of Wanda and our method:

• Similarity:
  1. Both Wanda and our method aim to minimize the discrepancy between the outputs of the pruned network and the original model under sparsity constraints.
• Difference:
  1. Wanda’s pruning metric $|W| \|X\|_2$ is heuristically designed, while we integrate $\\|W^* X^* - WX\\|_F$ into our optimization model to minimize the real error in a systematic and optimization-driven manner.
  2. Wanda’s $|W| \\|X\\|_2$ directly prunes weights without the opportunity to update the remaining weights for further error minimization, whereas our method not only enforces sparsity but also updates the remaining weights at the same time to minimize the error.

As shown in the results above, we summarize the PPL (lower is better) comparison across different sparsity levels as follows:

• 5% and 10%: intra- and inter- layer error correction $<$ intra-layer error correction only $<$ no error correction;
• 20%: intra-layer error correction only $<$ intra- and inter- layer error correction $<$ no error correction;
• 50%: intra-layer error correction only $<$ no error correction $<$ intra- and inter- layer error correction.

First, the results confirm the effectiveness of our intra-layer error correction mechanism, as it consistently outperforms the no-error-correction approach.

Second, the results confirm the effectiveness of using both intra- and inter-layer error correction at low sparsity levels, as it consistently outperforms the intra-layer error correction alone at 5% and 10% sparsity.

Third, the results show that using both intra- and inter-layer error correction is sensitive to sparsity levels and tends to perform worse at higher sparsity. Specifically, at 20% sparsity, it underperforms compared to intra-layer error correction alone, and at 50% sparsity, it even performs worse than the no-error-correction approach.

To explain why the use of both intra- and inter-layer error correction is sensitive to sparsity levels, we believe this occurs because higher sparsity levels make the pruning task more difficult, leading to greater error accumulation across layers. When both intra- and inter-layer error correction are applied, mitigating the accumulated error from previous layers may dominate the optimization objective in deeper layers, causing the pruning performance of the current layer to suffer.

Mathematically, let $W_k$ and $X_k$ represent the weight matrix and the activation of the $k$-th layer in the original network, respectively. Similarly, let $W_k^*$ and $X_k^*$ denote the pruned weight matrix and the corresponding activation in the pruned network. In a layer-wise pruning scheme with both intra- and inter-layer error correction mechanisms, we minimize the loss for each layer individually:

$X_k$ depends on the activation from the previous layer:

where $f_k$ represents some operations (e.g., activation function or normalization). Therefore, we can express the pruned activations recursively as:

The error at layer $k$ is defined as:

Therefore, under high sparsity level, this amplification often results in the accumulated error $\Delta X_k$ becoming dominant at deeper layers.

Thus, for large $k$, considering both intra- and inter-layer error correction mechanism, there is:

As a result, the optimization process shifts focus towards correcting this accumulated error rather than pruning the current weight matrix $W_k$.

In other words, minimizing the term (14) primarily addresses the error correction from previous layers rather than properly pruning the weight matrix $W_k$, which negatively impacts the pruning performance in deeper layers.

Response to Major Weaknesses:

1. Detailed Explanation of Intra-layer Error Correction Mechanism.

Thank you for your comment and interest in our intra-layer error correction mechanism. Our primary goal is to minimize discrepancies between the pruned and original models by adjusting weights after pruning in a layer-wise manner. We would like to provide a detailed explanation of this mechanism here and update the manuscript accordingly.

Denoting the pruned counterpart by $W^* \in \mathbb{R}^{m \times n}$, a straightforward approach to quantify the output error is to use the Frobenius norm of the difference between the outputs from the dense and pruned weights

which serves as a metric of the pruning quality at the target sparsity level and is widely adopted by work such as SparseGPT.

However, we observe that applying Equation (1) can lead to an error accumulation issue across sequential operators, as illustrated in Figure 2 of our manuscript. In the figure, $W_1$ and $W_2$ represent the weights of two sequential operators. Although Equation (1) effectively quantifies the output error between $W_1$ and its pruned counterpart $W_1^*$ since they are at the top of the layer and share the same inputs, issues arise when applying the same metric to the outputs of $W_2$ and $W_2^*$. Following Equation (1), the deviation between the outputs of $W_2$ and $W_2^*$ is computed with the same input $W_1X$. However, in a pruned network, the actual input for $W_2^*$ is $W_1^* X$, creating a discrepancy with $W_1X$ and thus leading to error propagation through the operators. To address this, we propose a method to sequentially prune weights within each pruning unit (e.g., a decoder layer of a Transformer), updating Equation (1) to:

where $X^*$ represents the input feature activation for $W^*$ from the sequentially pruned network.

Response to Minor Question:

Thank you for your question. We apply only the intra-layer error correction mechanism for two reasons:

1. Parallelization: Intra-layer error correction enables independent pruning of each decoder layer, allowing us to distribute the pruning task across multiple devices by assigning different decoder layers to different devices. This will increase the overall pruning efficiency.
2. Sparsity Sensitivity: While combining intra- and inter-layer error correction could intuitively reduce error accumulation across the network, we found that this approach is effective only at low sparsity levels. When the pruning task becomes harder (i.e., higher sparsity), global error correction tends to overshadow the pruning process of individual layers, ultimately leading to worse performance.

The first reason is straightforward; we will explain the second reason in more detail below.

We conducted a series of comparison experiments on OPT-125M at sparsity levels of 5%, 10%, 20%, and 50%. The experiments included three conditions: intra-layer error correction only, both intra- and inter-layer error correction, and no error correction. The results are presented in the following tables.

2. The Impact of Integration with Other Methods.

Regarding your question about the impact of integrating the proposed intra-layer error correction mechanism with other pruning methods, we would like to clarify that its effect depends on whether the layer-wise pruning methods include a "weight update" stage.

For methods without a "weight update" stage, such as Magnitude Pruning and Wanda, weights are directly removed based on a pruning metric. As a result, there is no opportunity to mitigate the error introduced by pruning through updating remaining weights, nor to apply our intra-layer error correction mechanism.

For methods with a "weight update" stage, such as SparseGPT, integration with our intra-layer error correction mechanism is feasible. To apply this mechanism, we need to update their error measurement from Equation (1) to Equation (2) and derive the corresponding mathematics. Here, we use SparseGPT as an example to demonstrate how to apply this mechanism. In the following, we write $W_i$ to denote the $i$-th row of the weight matrix $W$ and $\delta w$ to denote the pruning and weight updates on the row. It suffices to focus on a single row as the problems between different rows are independent.

Given Equation (5), we write the Lagrangian function as follows:

Then, we have

Solving this system, we have

Denote

then, we have

In summary, to apply our mechanism in SparseGPT, we first use Equation (9) as the new pruning metric to determine which elements to prune, and then apply Equation (7) to update the remaining weights.

Although the derivation is mathematically sound, implementing it within SparseGPT's framework presents significant challenges. SparseGPT’s pruning approach is designed for computational efficiency, relying on block-wise pruning and approximations of the Hessian matrix and its inverse. Integrating our correction mechanism necessitates additional matrix computations, which could compromise these efficiency goals. Despite the computational demands, we modified SparseGPT's code and evaluated its performance on OPT-125M and LLaMA-3-8B models using the WikiText dataset under 50% sparsity. The results are as follows:

These results validate the effectiveness of our intra-layer error correction mechanism and the FISTAPruner method.

Response to Weakness 1:

Thank you for your comment. We would like to clarify that FISTAPruner does not involve retraining during pruning; instead, it utilizes a layer-wise pruning approach, similar to methods like SparseGPT and Wanda. To clarify, retraining typically refers to the process of re-optimizing the entire model, usually by fine-tuning it on a specific loss function with respect to a training dataset after pruning. In contrast, FISTAPruner prunes the model in a layer-wise manner without such re-optimization, making it comparable to SparseGPT and Wanda, which also perform layer-wise pruning without retraining.

We understand the concern about computational costs, particularly in comparison with heuristic-based pruning methods like SparseGPT, Wanda, and DSnoT. While FISTAPruner may require slightly more pruning time due to its optimization-based approach, its key contribution is the introduction of an optimization model that systematically addresses the post-training pruning problem, leading to significantly better performance. This performance gain comes with an associated increase in pruning time, but the trade-off is justified by the improved outcomes.

Furthermore, while PERP (Parameter-Efficient Retraining after Pruning) is used for retraining pruned models, we have included comparisons between FISTAPruner and SparseGPT+PERP, as well as Wanda+PERP, in Table 4. The results show that FISTAPruner, which does not involve any retraining, outperforms SparseGPT and Wanda with retraining via PERP. Additionally, our method is compatible with retraining approaches and can provide a better initialization point for the retraining process, should that be required.

In conclusion, although FISTAPruner may take more time for pruning compared to heuristic methods, the substantial performance improvements it provides justify this additional time cost. Moreover, since our method does not involve retraining, the overall time cost remains competitive. We believe that, given the enhanced performance and the potential for further optimization, the time cost should not be viewed as a major concern.

Response to Weakness 2:

Thank you for your comment. As mentioned in Lines 188--190, our model incorporates $l_1$-norm regularization to encourage sparsity. Mathematically, the $l_1$-norm is non-smooth, which prevents the direct application of traditional gradient descent methods for optimization. Furthermore, although the sub-gradient descent method could be used, its convergence rate is $\mathcal{O}(1/\sqrt{k})$, which is slower compared to FISTA, which achieves a faster convergence rate of $\mathcal{O}(1/k^2)$, as mentioned in Lines 229--231.

Response to Question 1:

Thank you for your question. The key difference between retraining and our approach lies in the optimization objective. Retraining typically involves optimizing a model using a loss function that depends on the model's predictions, which is highly non-convex and requires computationally expensive backpropagation to compute gradients. In contrast, our approach uses a layer-wise error measured by the Frobenius norm in the objective function. This error is convex, and its gradient has closed-form expressions, making the optimization both convex and computationally more efficient. Therefore, while both methods aim to adjust the model, our approach does not involve retraining the model in the traditional sense of fine-tuning using a loss function.

Response to Question 2:

Thank you for your question. We indeed conduct sequential pruning within each decoder layer, specifically among the linear operators (K, Q, V, Out projections, and MLPs), to mitigate error accumulation through our intra-layer error correction mechanism. However, to enhance pruning efficiency, we treat each decoder layer as an independent pruning unit, enabling parallel pruning across multiple decoder layers on different devices. In other words, while pruning is performed sequentially within each individual decoder layer, the pruning process can occur in parallel across different layers, which allows us to strike a balance between performance and efficiency. We hope this clarifies how parallel pruning is achieved alongside the sequential pruning within layers.

We appreciate your thoughtful comments and will ensure that all the concerns and questions raised, including the clarification of the pruning process, are addressed more clearly in the revised version of the paper.

Dear Reviewer 68Xw,

Hi, as the discussion period is set to end soon, we kindly ask if you have any further questions or comments, or if there are any aspects of the paper that require additional clarification. Your feedback is invaluable to us, and we want to ensure we address any remaining concerns.

If there are no further concerns, we would greatly appreciate it if you could kindly update your score accordingly.

Best regards,
Authors 8547

Thank you for your comment regarding the work "Fast and Effective Weight Update for Pruned Large Language Models" (referred to as ADMM). We acknowledge this contribution and have included it in the related work section of our paper.

However, it is important to highlight that there are differences in the experimental environments (e.g. different versions of PyTorch, different GPUs, etc.) between our study and the ADMM framework. In our tests, the performance of the baseline methods for most models is notably lower (with higher perplexity results) than that reported in ADMM. For instance, in our environment, SparseGPT and Wanda yield perplexity of 6.54 and 6.46, respectively, for LLaMA-2-7B, compared to 6.51 and 6.42 in ADMM’s environment. Thus we suspect that the results for the methods proposed in ADMM are taking advantage of the testing environment compared to ours. This discrepancy suggests that direct performance comparisons may not be entirely fair.

Furthermore, even under these conditions, FISTAPruner outperforms ADMM across all other models reported, with the exception of LLaMA-2-7B, where their performances are quite similar. We believe this context is crucial for a comprehensive evaluation of the algorithms’ effectiveness.