We appreciate the reviewer for acknowledging the strength of our paper! Below we address the questions regarding 2:4 ratio, semi-structured benefits, assumption of the theoretical proof, ALM and EnumALM as well as model size justification.

W1:The focus on 2:4 pruning sparsity: 2:4 pruning is the most practical semi-structured sparsity pattern and the only one currently supported by commercial hardware

We thank the reviewer for the consideration! In Appendix G, we have more discussion on the practical relevance of the 2:4 sparsity pattern and the extensibility of ProxSparse. We focus on the 2:4 sparsity pattern in this paper because it is the most practical semi-structured format and the only one currently supported by commercial hardware. To the best of our knowledge, existing hardware such as NVIDIA Ampere GPUs only support 2:4 sparsity [1]. ProxSparse aligns directly with this hardware feature, making it readily applicable to real-world use cases.

W2:Why semi-structured pruning considering its lack of flexibility: Semi-structured pruning strikes a balance between efficiency and accuracy, and also benefiting from real hardware support.

The reviewer is correct that compared to unstructured pruning, semi-structured method cast more constraints, which is less flexible. However, unstructured pruning often does not directly translate to faster inference because it induces irregular access, and modern hardware exploits regularities in computation for faster speed. On the other hand, sturctured pruning typically offers the highest efficiency but suffers from huge accuracy loss due to its rigid constraints.

Semi-structured pruning is an important problem to study [2][3] in pruning, as it strikes a balance between efficiency and accuracy. A key benefit is its direct support on commercial hardware like NVIDIA Ampere GPUs and high-performance libraries for sparse operator, enabling real-world speedups. In this paper, we focus on semi-structured pruning and propose a relaxed end-to-end mask selection approach to identify optimal pruning masks for LLMs.

W3: relationship between ALM ans EnumALM: ALM is the subrouting of the EnumALM.

Thanks for the question! ALM is a subroutine used within EnumALM. Specifically, EnumALM solves the proximal operator for the 2:4 regularizer by enumerating and evaluating three candidate sparsity patterns: a 2-sparse solution (selected directly by top-k), a 3-sparse solution, and a dense (4-sparse) solution. For the latter two cases (3-sparse and 4-sparse), EnumALM invokes ALM to efficiently solve the corresponding convex subproblems with convergence guarantees.

W4: assumption of the proof: in general cases, our assumption holds because the "ReLU is weakly differentiable" and "weights are bounded", as explained below.

We thank the reviewer for the discussion of the assumptions! We acknowledge that the convergence analysis assumes the loss function is continuously differentiable and the weights remain bounded during optimization.

While the use of ReLU in the loss may technically violate differentiability at a single point (zero), this is a well-known and standard issue in deep learning. In practice, ReLU is differentiable almost everywhere, which is typically sufficient for convergence analyses in nonconvex optimization. Moreover, the population loss, as an expectation over a smooth data distribution, can remain continuously differentiable even when ReLU is used.

The other assumption—that the weights remain bounded—is rather mild and commonly used in convergence analyses. It is satisfied as long as the optimization does not diverge, which can often be ensured by using a sufficiently small learning rate. In our case, we observe stable behavior throughout, supporting the validity of this assumption. Meanwhile, ProxSparse's consistent performance across a variety of LLMs and tasks further supports the practical utility of our method.

W4: Larger model size experiment: we apologize for the current resource infeasibility on running experiments on larger (>30B) model. Our experiments show consistent good performance across different size of model.

We apologize for the lack of models larger than 30B due to limitations in our current resources. Notably, prior work on learning based LLM pruning [3] has also only conducted experiments on models up to ~15B in size. Nevertheless, our paper includes results on 14B model to demonstrate the effectiveness of our method. Across various model sizes, our approach consistently outperforms other baselines, highlighting its robustness and applicability.

[1] NVIDIA AMPERE GA102 GPU ARCHITECTURE [2]Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [3]MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models

We thank the reviewer for the valuable feedback! Below we share responses on comparison w/ layer-wise method and MaskLLM, inference efficiency.

W1:Comparing newer layer-wise method: We achieve better results than OWL and AlphaPrune

OWL[1] and AlphaPrune[2] are important works in pruning, aiming to determine layer-specfic ratio to protect important layers. We are happy to discuss them in our paper!

In the meantime, we would like to respectfully argue that they are not very well-suited in semi-structured pruning, as the sparse operator supported by hardware typically requires all blocks to strictly adhere the pattern, making applying varying ratios hard.

Nevertheless, we conduct more experiments on AlphaPrune and OWL for comparison. We follow mixed sparsity proposed in OWL and AlphaPrune with Wanda, that layers can have varying ratios, while the overall ratio remains 2:4. We see ProxSparse outperforms OWL and Alphaprune on Anon.Model-1 and Mistral on PPL and acc, showing the strength of our end-to-end optimization. Further, as pruning patterns become more fine-grained (e.g., 2:4), varying layer-wise pruning ratios become less effective as critical weights might still be removed within each block. This was reported in both paper, where 4:8 pruning performed just similarly to uniform pruning in Wanda. This highlights the benefits of ProxSparse in identifying fine-grained semi-structured masks.

We have more baseline (ADMMPrune) discussion as proposed by reviewer hyi6. ProxSparse achieves better results compared to ADMMPrune. Please kindly refer here for more details!

Table 2: Comparison with OWL, AlphaPrune and ProxSparse on Mistral-v0.3-7b and Anon.Model-1. ProxSparse achieves the best.

W2:Comparison w/ MaskLLM: Complementary method with fundamentally different design, we excel in low-scale regime.

Thanks for the question! We note that ProxSparse and MaskLLM differs fundamentally in approaching semi-structured mask, and we view MaskLLM to be a complementary as it focuses on larger data regime

• Mechanism difference: MaskLLM and ProxSparse take fundamentally different ways in pruning. We both tackle the non-differentiable task of selecting N out of M per block. MaskLLM sidesteps this with a probabilistic sampling approach, learning to sample the correct weights. In contrast, ProxSparse relaxes the hard constraint into a smooth optimization and performs optimization via proximal gradient descent, with theoretically provable properties for the proposed 2:4 regularizer.

• Larger batch exps: | Method | 1024 | 2048 | |-------------|------|------| | MaskLLM | 10| 9.5 | | SparseGPT | 10.18| 10.16| | Wanda | 11.38| 11.4| | ProxSparse | 8.38 | 8.23 |

Table 3: PPL on Anon.Model-1 on extended sample size. ProxSparse achieves the best

Table 3 evaluates ProxSparse with larger sample size. In this small-scale data regime, ProxSparse outperforms all baselines, demonstrating its superiority. We note that our targeted low-scale calibration regime is practical in LLM contexts, making our method more accessible in the real world.

W3:Real world efficiency: ProxSparse leads to 1.35x inference speedup and 37.3% reduced peak memory usage

Thanks for the comments! As discussed in Appendix H, our analysis demonstrates that, beyond reduced FLOPs, ProxSparse achieves 1.35× inference speedup and 37.3% reduction in peak memory usage. These results highlight the practical efficiency gains enabled with semi-structured sparsity induced by ProxSparse.

Thanks again!

[1]Yin et al. OWL [2]Lu et al. AlphaPruning

We thank the reviewer for the insightful comments! Below, we address several questions raised including sparsity pattern selection, Hessian-based pruning, anonymized citations, and the anonymized model family.

W1: The focus on 2:4 pruning sparsity: 2:4 pruning is the most practical semi-structured sparsity pattern, and the only pattern currently supported by commercial hardware

We thank the reviewer for the consideration! In Appendix G in our paper, we have more discussion on the practical scenario of the 2:4 sparsity pattern and the extensibility of ProxSparse. We focus on the 2:4 sparsity pattern in this paper because it is the most practical semi-structured format and the only one currently supported by commercial hardware. To the best of our knowledge, existing hardware such as NVIDIA Ampere GPUs only support 2:4 sparsity [1]. ProxSparse aligns directly with this hardware feature, making it readily applicable to real-world use cases.

W2: A misunderstanding on Why Hessian based method is local-wise: Our claim is localized pruning (i.e. w/ Hessian) hinder the pruning. Meanwhile, the heavy hessian computation makes it hard for global optimization.

We thank the reviewer for raising the great point! We would like to first clarify on our claim, that we argue previous layer-wised pruning utilizes hessian metrics [2] or per-output based importance score [3] fail to select mask well because of the localized information constraints. Our method enables an end-to-end pruning mechanism that makes the pruning having well-informed solution.

In the meantime, while it is true as mentioned in [4], that Hessian can be evaluated through the global loss. We note that this is impractical in pruning LLM. Even in [4], which focuses on smaller CNN model, the authors report using Hessian incurs 30x inefficiency, leading to huge overhead. For an LLM with enormous parameter size (~B), it is even harder to calculate the Hessian, let alone to compute it round by round in end-to-end optimization. As further evidence, SparseGPT specifically highlights the computational burden of Hessian estimation. To overcome this, it proposes a Fast Approximate Reconstruction method to simulate Hessian more efficiently. Yet it is still limited in layer-wised pruning. In contrast, our proposed end-to-end optimization scheme delievers better performance compare against those layer-wised pruning methods.

W3: Anonymous citation: a private communication and we will update it upon acceptance

The anonymous citation is a private communication currently under a double-blind submission policy. We have provided detailed discussions of it in Section 3 and confirm that we will update the it upon acceptance.

W4: Anonymous model family: We anonymized some models (Anon.model-1,2,3) due to IP constraints, and we believe the 7 models presented have broad coverage and ProxSparse exhibits consistent trends.

We appriciate the reviewer for the understanding! We didn't reveal the anonymous model family name due to internal policy constraints. However, we note that these anonymized models are among the most competitive LLMs, as evidenced with good PPL and accuracy in our benchmarks. We hope those models serve as additional data points to further support the effectiveness of our method. Meanwhile, we believe our experiments offer broad coverage on top-performing models, including Mistral, OpenLlama and Qwen family, plus the anonymous model family. The consistent results highlight the superiority of our method, which is robust and widely applicable to the top-tier LLMs.

Thanks again!

[1] NVIDIA AMPERE GA102 GPU ARCHITECTURE [2]SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot [3]A SIMPLE AND EFFECTIVE PRUNING APPROACH FOR LARGE LANGUAGE MODELS [4]Pruning Convolutional Neural Networks for Resource Efficient Inference. ICLR 2017

We thank the reviewer for recognizing the effectiveness and practicality of ProxSparse! Below, we address the questions raised regarding Llama, ADMMPrune comparisons, clarification on weight updates and MaskLLM comparison.

W1: Lack of Llama results: We anonymized Anon.model-1,2,3 due to IP constraints, and we believe the 7 models presented have broad coverage and ProxSparse exhibits consistent trends.

We didn't reveal the anonymous model family name due to internal policy constraints. However, we note that these anonymized models are among the most competitive LLMs, as evidenced with good PPL and accuracy in our benchmarks. We hope those models serve as additional data points to further support the effectiveness of our method. We appriciate the reviewer for the understanding! Meanwhile, we believe our experiments offer broad coverage on top-performing models, including Mistral, OpenLlama and Qwen family, plus the anonymous model family. The consistent results highlight the superiority of our method, which is robust and widely applicable to the top-tier LLMs.

W2: Comparison with ADMMPrune: we achieve better results!

We appreiciate the reviewer's notes on newer baseline named ADMMPrune[1]! We benchmarked ADMMPrune to compare with ProxSparse on Anon.Model-1 and Mistral model, using same evaluation settings discussed in paper. As shown, ProxSparse outperforms ADMMPrune in both models, achieving lower PPL (8.51 vs. 9.67) and higher acc (47.6% vs. 45.5%), highlighting its effectiveness. We attribute the superority of ProxSparse to its end-to-end optimization process, which goes beyond solely relying on local layer-wised information. We are happy to discuss ADMMPrune in our revised paper!

In the meantime, we include more baselines (OWL and AlphaPrune) discussion as proposed by reviewer 2EPP. ProxSparse consistently achieves better performance on them. Please kindly refer here for more details!

Table 1: Comparison between ADMMPrune and ProxSparse on Mistral-v0.3-7b and Anon.Model-1. ProxSparse consistently achieves better performance

W3: Clarification on weight update: our method does NOT update the unpruned LLM parameters.

ProxSparse is an learned method that identifies the optimal mask without further weight updates on retained weights. In other words, the retained weight after applying ProxSparse-selected mask remains identical to its initialization, similar to Wanda and magnitude prune. In ProxSparse, the end-to-end optimization with calibration data is sololy used to determine the mask. Even when compared to SparseGPT and ADMMPrune—which update weights after pruning—ProxSparse consistently achieves higher performance, underscoring its effectiveness in identifying high-quality pruning masks.

W4: Comparison with MaskLLM on effectiveness: ProxSparse consumes 25x less tokens compared to MaskLLM.

We are happy to provide more intutive and direct comparison between MaskLLM and ProxSparse. As discussed in our paper, MaskLLM employs a fundamentally different design with Gumbel Softmax sampling for learning the mask. We view MaskLLM as a complementary to ours as it focuses on large sample regimes, whereas ProxSparse operates effectively with much smaller sample size. This makes ProxSparse more practical and accessible in LLM era.

Here we present direct comparison betwen MaskLLM and ProxSparse on Anon.Model-2. In terms of consumed tokens, ProxSparse achieves a PPL of 8.51 with 1638400 tokens, outperforming Wanda (11.42), SparseGPT (10.298) and MaskLLM (11). For MaskLLM to achieve a comparable PPL, it consumes 40960000 tokens, 25x larger than ProxSparse. This, in general, demonstrates the superiority of our pruning method with small-scale calibration data like ADMMPrune, Wanda and SparseGPT.

Thanks again!

[1]ADMMPrune:Fast and optimal weight update for pruned large language models