We authors greatly thank the reviewer for constructive comments on this work. We would like to clarify the following points:

W1: Evaluation Scope: Lacks zero-shot and few-shot task evaluations (cf. Wanda’s Tables 2, 21), limiting practical relevance.
Q3: Zero-shot Task Evaluation: Why were zero-shot accuracies (e.g., as in Wanda’s Table 2, also Tables in the appendix e.g., Table 23) not included? Adding these could align your evaluation with Wanda’s, enhancing practical relevance. A justification or additional results would influence my view on the method’s applicability.

Below, we provide additional results on more benchmarks.

Table: Performance of Llama-7B on three additional benchmarks, along with the results from Table 3.

W2: Missing details on calibration data and iteration specifics (e.g., ( T ) selection).

1. The proposed iterative refinement method is entirely data-free and does not require calibration data. As shown in Algorithm 1, the only inputs are:

• Dense weight matrix $W$
• Binary mask $P$ (from pruning)
• Target rank $k$
• Number of iterations $T$

We use WikiText-2 for perplexity evaluation and 'allenai/c4' as the calibration data for Wanda pruning as well as Wanda + Ours.

2. We consistently uses T=50 across experiments, which is sufficient for achieving most of the potential error reduction while maintaining computational efficiency. Since the overall time complexity is $O(T \cdot min(mn^2, m^2n))$, where $m$, $n$ are the number of rows and columns of $W$.

W3: Omits SparseGPT and fine-tuning baselines, reducing comparative strength against Wanda..
Q4: SparseGPT Comparison: Why was SparseGPT excluded from experimental comparisons despite its relevance (cf. Wanda’s Table 3)? Including it could better position your method; its absence weakens the competitive analysis.

Thank you for the suggestion. We provide the results of SparseGPT 2:4 and SparseGPT 2:4 + Ours in the following table.

Table: Performance of SparseGPT 2:4 and SparseGPT 2:4 + Ours using Llama-7B on three additional benchmarks, along with the results from Table 3.

Q1: Calibration Data Details: What calibration data (size, source) was used for perplexity experiments? Wanda specifies 128 C4 sequences; this omission affects reproducibility. A response detailing this would strengthen confidence in the results.

Thank you for your suggestion, we will add these details in revised version. We also use 128 C4 sequences as the calibration data for Wanda pruning as well as Wanda + Ours. For perplexity evaluation, we use 128 sequences from WikiText-2 dataset.

Q2: Performance Target for Parameter Reduction: The abstract claims an 8.6% parameter reduction for a specific target at 50% sparsity. What is this target, and can you provide supporting data? Without this, the claim feels unsubstantiated.

The performance target is the perplexity metric on WikiText-2 dataset, we will revise the abstract to clarify this. We provide the detailed results in Figure 3(a).

Thank you for your time and effort in reviewing our paper. We appreciate your constructive feedback and suggestions.

W1 (Computational Complexity Analysis): While the paper discusses parameter efficiency of low-rank refinement, it does not thoroughly analyze the computational cost of the iterative update algorithm compared to alternative approaches such as the optimization-based PCP method.

The primary computational bottleneck in the proposed algorithm is the SVD computation performed in each iteration (line 171, page 4). For a weight matrix $\mathbf{W} \in \mathbb{R}^{m \times n}$, the time complexity of SVD is $O(mn^2)$ (assuming $m \geq n$). With $T$ iterations, the overall time complexity is $O(Tmn^2)$.

In contrast, the PCP baseline would typically use an iterative optimization method (like Adam) that also requires SVD computations in each iteration to compute the nuclear norm. Furthermore, the PCP baseline requires substantially more iterations ($T=5000$) to achieve comparable results to the proposed method ($T=50$).

Table: Computational Efficiency Comparison

W2 (Inference Latency and Memory Consumption): The proposed method is theoretically hardware-efficient as it enables structured N:M pruning. However, the inference latency and memory consumption of low-rank refinement is not measured.

Below, we provide some missing measurements and analyses. Note that we deliberately employed a conservative evaluation approach to ensure fair comparison with the dense baseline model. We maintained all matrices in their dense representation format, preserving zero elements rather than utilizing sparse matrix formats or specialized hardware acceleration.

Table: Parameter Count and Evaluation Time for Complete ARC-Challenge Benchmark.

Q1: The rank k is manually set in the paper, will a poor choice of k lead to unnecessary computational overhead or insufficient recovery of pruned weights?

Yes. The choice of k leads to a trade-off between the performance recovery and the computational cost. The low-rank component adds $k(m+n)$ parameters and $2k(m+n)$ FLOPs. Therefore, an unnecessarily high k value directly increases these costs with diminishing returns. Figure 2(b) in the paper shows the cumulative energy retention for different k values, and the curve begins to flatten as k increases, indicating diminishing returns.

Q2: If given fixed inputs, does the iterative refinement method always converge to a stable solution?

Yes. As shown in Theorem 4.2, Theorem 4.3 and Corollary 4.4 in the manuscript, the iterative refinement method always converges to a stable solution as $T \to \infty$ and the approximation error is bounded by:

\left\\|W - \left(S^{(t)} + L_k^{(t)}\right)\right\\|_F \leq \sqrt{(r-k)} \sigma\_{k+1}\left(L^{(t)}\right),

where $r$ is the rank of the weight matrix $W$, $\sigma_{k+1}\left(L^{(t)}\right)$ is the $(k+1)$-th largest singular value of $L^{(t)}$.

W1: While the paper mentions magnitude pruning and N:M structured pruning, it does not discuss structured sparsity techniques, such as block sparsity or channel pruning, which have been shown to improve hardware efficiency and model performance.

We thank the reviewer for pointing out the importance of structured sparsity techniques. It is straightforward to apply the proposed method to structured sparsity techniques such as block sparsity or channel pruning by by simply changing the binary mask matrix P. The core algorithm is compatible with any arbitrary binary mask pattern, including structured ones. For structured patterns like block sparsity (e.g., 4×4 blocks) or channel pruning, P would have the corresponding pattern of 0s and 1s.

Theoretically, Theorem 4.1 (Sparsity Preservation) guarantees that the method preserves whatever sparsity pattern is defined by $P$.

W2: This paper does not include the full details of lemma A4.2 in the Appendix A.

The proof of Lemma A4.2 currently appears before the lemma itself. In the revised version, we will relocate the proof to its proper position following Lemma A4.2.

proof of Lemma A4.2: Let $m_{ij}$ and $p_{ij}$ be the elements of $M$ and $P$ respectively. By definition of the Frobenius inner product and element-wise multiplication:

$$\langle M, P \odot M \rangle_F = \sum_{i=1}^m \sum_{j=1}^n m_{ij} (p_{ij} m_{ij}) = \sum_{i=1}^m \sum_{j=1}^n p_{ij} m_{ij}^2\leq \sum_{i=1}^m \sum_{j=1}^n m_{ij}^2 = \langle M, M \rangle_F$$

W3: The iterative refinement process introduces new hyperparameters (e.g., rank k, number of iterations T), but the paper does not provide a clear guideline on how these should be selected across different models and sparsity levels.

• Rank parameter k: We consistently set $k=128$ across the experiments to demonstrate the effectiveness of low-rank refinement. In Figure 4(a), we visualize the singular value spectrum of the residual matrix $L=W-S'$ for different values of $k$ (from 64 to 512) and $T=50$. As $k$ increases from 64 to 512, we see higher magnitudes for the top singular values. This indicates that larger $k$ values allow the method to retain more information. But when $k$ is too large, the performance gain diminishes and the computational cost increases. Therefore, the choice of $k$ should strike a balance between the performance and the computational cost.

• Number of iterations T: The primary computational bottleneck in the proposed algorithm is the SVD computation performed in each iteration (line 171, page 4). For a weight matrix $\mathbf{W} \in \mathbb{R}^{m \times n}$, the time complexity of SVD is $O(mn^2)$ (assuming $m \geq n$). With $T$ iterations, the overall time complexity is $O(Tmn^2)$. On the other hand, as shown in Figure 4(d) and the diminishing convergence speed as stated by Theorem 4.5 (Error Bound), $T=50$ appears to be a reasonable choice for the number of iterations and further iterations yield diminishing returns in terms of error reduction.

W4: The computational cost of iterative updates may be high as the size of the weight matrix increases, which may limit the applicability of the method to very large models.

As the size of the weight matrix increases, the computational cost of the proposed method increases accordingly. The primary computational bottleneck in the proposed algorithm is the SVD computation performed in each iteration (line 171, page 4). For a weight matrix $\mathbf{W} \in \mathbb{R}^{m \times n}$, the time complexity of SVD is $O(mn^2)$ (assuming $m \geq n$). With $T$ iterations, the overall time complexity is $O(Tmn^2)$.

In contrast, the PCP baseline would typically use an iterative optimization method (like Adam) that also requires SVD computations in each iteration to compute the nuclear norm. Furthermore, the PCP baseline requires substantially more iterations ($T=5000$) to achieve comparable results to the proposed method ($T=50$).

Table: Computational Efficiency Comparison

Q1: In line 6 of Algorithm 1 $r(t) = 1 + (k-1) / (T-1)$, what is the intuition behind this equation?

This equation in Algorithm 1 defines a linear schedule for increasing the rank from 2 to k across T iterations. Starting with low rank forces the algorithm to capture the most important singular components first, and each iteration gradually incorporates more subtle details from higher singular components. The gradual increase in rank helps maintain numerical stability and helps the algorithm to converge faster.

Q2: How to select the hyperparameters? See weaknesses 3.

Please refer to the response to W3.

Thank you for the insightful feedback and constructive suggestions.

W1: End-to-end inference acceleration is missing. It's better to report speedup for completeness.

1. Without sparse matrix formats or specialized hardware acceleration, the inference time is as follows:

Firstly, for a fair comparison with the dense baseline model, we took a conservative approach in our evaluation. Specifically, we stored all matrices in their full dense format and kept all zero elements (the weight matrices are torch.Tensor objects), without leveraging any sparse matrix formats or hardware-specific optimizations for acceleration.

Table: Parameter Count and Evaluation Time for Complete ARC-Challenge Benchmark.

2. With sparse matrix formats and hardware acceleration, we convert the weight matrices to their sparse format. Specifically, we use torch.sparse.to_sparse_semi_structured to convert the weight matrices to torch.sparse.SparseSemiStructuredTensor objects. The following table shows the memory usage of the proposed method with and without sparse matrix formats. The actual inference speedup varies based on factors like batch size and input sequence length. With hardware-accelerated N:M structured pruning (on NVIDIA Ampere GPUs and newer), we can observe approximately ~1.1x faster inference in wall-clock time compared to dense models. However, it's important to note that for unstructured pruning patterns or on GPUs without dedicated sparse acceleration support, using sparse operations can actually result in slower inference times compared to dense computation.

Table: A comparison of memory usage with and without end-to-end inference acceleration.

W2: I suggest that the authors add add some frontier models, such as Llama 3.1 8B. Llama is quite old and may lack reasoning and mathematics abilities. This may affect the GSM8K results.

Thank you for the suggestion. We have added Llama-3.1-8B to the experiments. The results are shown in the following table.

Table: Llama-3.1-8B results.