Compress Large Language Models via Collaboration Between Learning and Matrix Approximation

W1: GPU acceleration.

Thanks for your suggestion. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results. To address this, we followed the ATP[r1] protocol and conducted GPU inference tests using nm-vllm. The results are presented in Table R1. Compared with unstructured methods, our approach, Low rank & Sparse, leads to more practically meaningful GPU inference acceleration.

Table R1 GPU inference speedup LLaMA2-13B, on a single 80 GB A100 GPU

W2: More justification about Motivation.

Thanks for your suggestion. We provide a detailed explanation of their rationale and implementation below.

Motivation of policy gradient. Our bilevel optimization problem in Eqn. (4) of the paper takes the form of

It involves a nested structure where the outer objective depends on the output of a non-differentiable inner procedure (specifically, RPCA decomposition of weight matrices).

Existing approaches to bilevel optimization (e.g., [r2, r3]) typically rely on implicit differentiation. As discussed in the submission's main text, if we were to follow the conventional approach using implicit differentiation, the resulting gradient estimate would involve second-order derivatives and take the following form:

However, in our case, the RPCA solution $\mathcal{W}(s, \kappa)$ does not admit a tractable or closed-form gradient, making $\nabla_{\theta} \mathcal{W}(s, \kappa)$ cannot be computed directly.

To overcome this challenge, we resort to policy gradient estimator, which allow us to estimate $\nabla_{\theta} \Phi(\theta)$ without calculating $\nabla_{\theta} \mathcal{W}(s, \kappa)$. The PGE is given by such a simpler form:

Note that explicitly computing the expectation above is infeasible, we use the policy gradient $\mathcal{L}_ {\mathcal{B}}(\mathcal{W}(s, \kappa))\nabla_{\theta} \log p(s, \kappa \mid \theta)$, which is an unbiased estimation of $\nabla_{\theta} \Phi(\theta)$. This is why we need and can use this stochastic approximation.

Motivation of truncated Gaussion. The truncated Gaussian prior is our novel design, which effectively mitigates the risk of gradient explosion (see Section 4.2). This issue can arise when using a standard Gaussian distribution, as its gradient contains a $1/\sigma^2$ term. To ensure convergence, one typically needs to reduce $\sigma$ to concentrate the sampling distribution, but this can lead to unstable gradients. The truncated version addresses this problem by allowing us to concentrate samples via truncation while keeping $\sigma$ fixed, thereby maintaining gradient stability.

W3: The use of diagonal approximation of $X^\top X$ and More Comparison with HASLLE-free.

We thank the reviewer for raising this important concern.

Reasons of using diagonal approximation. We would like to discuss the reasons why we use diagonal approximation:

• We adopt this approximation primarily to ensure a fair comparison. Since OATS, the strong baseline in our study, employs this diagonal approximation, we also follow this choice to isolate the superiority of our proposed collaborative allocation framework.

• The core contribution of our work is the collaborative allocation framework, instead of the specific approximation to $X^\top X$.

Effects of $X^\top X$. To address the concern, we conduct additional experiments by removing the diagonal approximation used in OATS and directly applying our bilevel framework on the unmodified low-rank sparse structure. The results in Table R2 below show that our method continues to deliver improvements, indicating that the proposed framework is robust and effective even without relying on this approximation.

Table R2 Comparison with OATS on Phi-3 mini 40% prune rate, with and without diag $X^\top X$.

Comparisons with HASSLE-free. According to the reviewer’s suggestion, we compare our method with the HASSLE-free approach. Since HASSLE-free adopts a fixed semi-structured sparsity pattern and a predefined rank, it differs from our adaptive sparsity allocation strategy. As a first step, we reproduced their configuration using our QR-based compression under the same fixed sparsity setting. While this led to slightly lower performance, we argue that such a comparison is not entirely fair, as fixed sparsity allocation is not the main contribution of our method. For a more meaningful evaluation, we further compared both methods under the same overall sparsity ratio (50%), using results directly reported in their original paper. In this setting, our method outperforms HASSLE-free, demonstrating the effectiveness of our adaptive allocation strategy.

Table R3 Comparison with HASSLE-free on LLaMA3-8B 50% prune rate

References

[r1] Lujun Li et al. "Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models."

[r2] Pedregosa. "Hyperparameter optimization with approximate gradient. "

[r3] Grazzi et al. "On the iteration complexity of hypergradient computation."

We sincerely appreciate your active engagement in the discussion and are happy to further address your concerns.

Q4: nm-vllm is outdated, your dense baselines are not as strong.

Thank you for your question. We acknowledge that nm-vllm has indeed not been updated for months. However, it remains one of the few libraries that support GPU-accelerated unstructured sparse inference, and has been adopted in recent research works like ATP[r1]. For these reasons, we have selected it as our benchmark.

We have verified that vLLM indeed has newer versions with better performance. However, the latest version has not yet been integrated into the nm-vllm framework. Therefore, using the latest vLLM for dense model inference as a baseline, while evaluating sparse acceleration within nm-vllm (which is based on an older vLLM version), leads to an inconsistent setup. As a result, such a setting cannot accurately reflect the actual speedup achieved by sparse acceleration methods.

To the best of our knowledge, there is currently no more appropriate library available for sparse acceleration on GPU. Thus, we think our evaluation setting is a reasonable and feasible choice under the current circumstances.

Q5: Are you using a single prompt to calculate the throughput? How many tokens are you generating?

Thank you for your question. We would like to clarify the following: Our testing employed num_requests=16 with a consistent max_token_length setting of 512. All experiments were conducted using the same configurations and were repeated 5 times to ensure fairness and accuracy in our results.

Q6: Have you implemented your own kernels?

Currently, there is a lack of GPU inference acceleration libraries specifically designed for low-rank + sparse matrix computation. We did not implement custom kernels from scratch either, although this is theoretically feasible, it would require substantial engineering effort, which is difficult to accomplish during the discussion phase.

Instead, our approach is as follows: we treat the sparse matrix as the backbone and leverage the nm-vllm engine to perform sparse inference accordingly. Meanwhile, the low-rank matrices are integrated as LoRA adapters during the forward pass. This design is well-supported by nm-vllm and has shown better practical speedup performance.

Q7: What is the justification for "Given the same total parameter count, part of our parameters are implemented as low-rank matrices, which are more hardware-friendly and better suited for achieving practical speedups."

Thank you for your question. Our statement is inferred from both our experimental observations and empirical insights from prior work in the field.

Due to the complexity of the nm-vllm engine, we have not yet conducted a systematic evaluation of the runtime efficiency of each individual operator within the model. However, from the observed acceleration behavior on CPUs, we believe that our assumption, that low-rank matrices offer better hardware efficiency under the same parameter budget, is reasonable.

We will provide a more comprehensive and detailed evaluation in the revised version.

Q8: You are still using a diagonal approximation, one that is slightly more accurate.

Thank you for your recognition. While this is not the core contribution of our paper, we agree that exploring more accurate inner-loop optimization using the complete full Hessian, beyond diagonal approximations, is a meaningful and promising direction. However, such an extension would require substantial modifications to the algorithm and is difficult to evaluate systematically within the limited discussion phase.

We will conduct a thorough evaluation and include the full results in the revised version.

We hope this explanation helps clarify your concerns.

Thank you very much for your positive evaluation and thoughtful suggestions. We sincerely appreciate your recognition of our work.

Quality W1: Results presentation.

We sincerely appreciate your suggestions. Due to space limitation, we present the experimental results under higher sparsity level in our response to reviewer AQTX's Table R1. All other suggestions will be taken in-detail in the revised verision if accepted.

Quality W2: GPU results.

Thanks for your suggestion. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results. To address this, we followed the ATP[r1] protocol and conducted GPU inference tests using nm-vllm. Due to the space limit, please refer to our response to reviewer AQTX's W6&Q1.

Quality W3: Hyperparameter sensitivity.

Thanks. We would like to clarify following facts to show that our algorithm is not sensitive to these hyperparameters.

• In fact, we tuned these hyperparameters on Phi-3 mini and applied the same configuration across all other model experiments. Besides, parameters such as the initialization of the truncated Gaussian mean $\mu$ and the number of inner-loop iterations are consistent with those used in OATS.
• We also conduct ablation studies on these key hyperparameters, and the results are presented in Table R1 below.

Table R1. Ablation study of hyperparameters Phi3-mini with 50% prune rate, wikitext2 PPL

Clarity W1: Experimental details.

We sincerely apologize for the omissions in the original submission and appreciate your insightful comments. We will ensure that these aspects are addressed thoroughly in the final version.

First, regarding Table 3(in submission), our goal is to isolate the effect of different components in our method—namely, sparsity allocation across layers and rank ratio across matrices. Specifically:

• “w/o Rank” refers to not performing learnable rank allocation; each matrix is assigned a fixed rank ratio, while the layer-wise sparsity is still learned.
• “w/o Sparsity” denotes the setting where compression rates are fixed uniformly across layers, and only the rank ratios of matrices are learned.
• “w/o Sparsity & Rank” disables both learnable sparsity and rank allocation. All layers and matrices are compressed with uniform configurations.

Second, we agree that more background on Wanda and SparseGPT would be beneficial, especially for readers less familiar with the literature. Specifically, we replaced the inner optimization in our bilevel framework—from an RPCA-based method using QR decomposition—to Wanda and SparseGPT, enabling the outer loop to learn the sparsity allocation for each layer. We will add a paragraph to introduce these methods in the revised version if accepted.

Originality W1: Heuristics' originality.

We apologize for not clarifying the originality and standardness of some of our design choices. We will include following clarifications in the revised version if accepted.

• First, subtracting a zero-mean and highly correlated control variate is a core idea in variance reduction techniques used in statistics and Monte Carlo methods. However, the specific design of variance reduction strategies can vary significantly across different applications. Our contribution lies in tailoring a variance reduction approach specifically for our bilevel optimization framework, which enables more stable policy gradient estimation.
• Second, the truncated Gaussian prior is our novel design, which effectively mitigates the risk of gradient explosion (see Section 4.2). This issue can arise when using a standard Gaussian distribution, as its gradient contains a $1/\sigma^2$ term. To ensure convergence, one typically needs to reduce $\sigma$ to concentrate the sampling distribution, but this can lead to unstable gradients. The truncated version addresses this problem by allowing us to concentrate samples via truncation while keeping $\sigma$ fixed, thereby maintaining gradient stability.
• Third, the cubic annealing schedule has been used in other applications, such as control the prune rate during training [r2]. We introduce it to our truncated Gaussian prior and find it works well.

Q1: Hyperparameters and RPCA convergence.

Regarding hyperparameters' sensitivity, please refer to our response to Quality W3.

Unlike conventional RPCA solvers that rely on accurate decomposition, our bilevel pruning framework only requires approximate gradients. The QR-based estimator strikes a balance between efficiency and performance. Empirical results confirm that this choice does not hinder optimization quality.

Table R2. Relative error(%) of q_proj and o_proj in Phi-3 mini's 1st Layer. The matrix o_proj has relative higher rank, leading to larger approximation error for both SVD and QR.

Q2: Compression time.

We acknowledge that, unlike one‑shot methods such as Wanda or SparseGPT, our bilevel framework incurs additional runtime. However, our entire process still completes within a few hours on standard hardware, which is always affordable in downstearm applications.

Importantly, model compression is a one‑time offline cost, where peak GPU memory usage often outweighs total runtime in practical deployments. Our method uses no more memory than one‑shot approaches but delivers superior accuracy and inference speedups, making the extra time investment worthwhile. The results of compression time are summarized in Table R3. Even QR, a simplified version of our method, still outperforms SparseGPT and Wanda under comparable time cost.

Table R3. Compression time, Phi-3 mini 60% prune rate

Q3: Sparsity allocation compared with OWL.

Due to the rebuttal constraint, we are unable to present the full comparison figure. Instead, we report the sparsity allocations of several selected layers in the table below. Compared with OWL, our method exhibits a similar overall trend—allocating more parameters to the early layers, applying stronger compression to the middle-to-late layers, and preserving more parameters in the final layer. This aligns with the general understanding of layer-wise redundancy in large language models. Notably, our method shows larger differences than OWL, more accurately reflecting the redundancy across layers and contributes to better overall performance.

Table R4. Sparsity allocation OWL & Ours Phi-3 mini 60% prune rate

Q4: Fully calibration-aware Optimization.

Thank you very much for your insightful suggestion regarding the improvement of our algorithm. After careful derivation, we confirm that the calibration-aware loss function you proposed can indeed be formulated and optimized. We present the derivation in detail below.

Here is a brief derivation of the refined optimization problem:

We still use alternating minimization strategy. In each iteration, we update $U$, $V$, and $S$ as follows:

where $A=XX^\top, B_{t+1}=(W-S_t)XX^\top$. Based on the derivation in the main text, we obtain the following update rules using QR decomposition:

We conduct experiment to evaluate this approach and report the results in Table R5. It shows that performance of these two methods are comparable. If the paper is accepted, we will include additional experiments in the revised version to evaluate the superiority of calibration-aware method.

Table R5. Experiment on Phi-3 mini 50% prune rate

References

[r1] Lujun Li et al. "Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models."

[r2] Zhu, M. H. et al. "To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression."

W1: Limited citations on quantization.

Thank you for this valuable suggestion. We fully agree that incorporating these recent works in the area of LLM compression will strengthen the contextualization and relevance of our work. We will include a paragraph reviewing these papers in the revised version if accepted.

W2 & Q1: Computational overhead and Scability.

Actually, our method has good performance on scalability due to the following reasons:

1. On memory cost. Since our pruning process does not require backpropagation, the computations can be performed in a layer-wise manner. Specifically, we load and process the model one layer at a time, which effectively prevents memory usage from scaling with the overall model size.

2. On computational cost. Thanks to the efficiency of our QR decomposition—over 75× faster than SVD—the computational cost for the inner loop is of the same order as a single forward pass. In all our experiments, we set the number of epochs to 1, making the overall cost affordable for most downstream practitioners, even when working with large-scale LLMs.

Results on Time Cost. Table R2 reports the time costs for QR decomposition, and other time (model (layer) loading, and GPU forward computation, etc.) per layer. The results show that these costs scale approximately linearly with the model size. This verifies our analysis above. Notice that loading and forward occupies most of the time cost, which can be improved by engineering optimization techniques such as DeepSpeed.

Table R1. Time taken of QR-based RPCA and other time

Results on LLaMA3-70B. Table R2 shows that our method can outperform the baseline OATS. More results will be included in the revised version if accepted.

Table R2. Results on LLaMA3-70B.

W3: Limited discussion of trade-offs.

We thank the reviewer for pointing out the lack of discussion on the trade-offs. In the revised version, we will include an additional section that explicitly analyzes the relationship among these factors.

Specifically, based on our findings and prior work, we observe that under sparsity levels below 50%, the compressed model retains most of its original performance while achieving meaningful acceleration on both CPU and GPU platforms. When the sparsity increases beyond this point, although the inference speed continues to improve, the performance degradation becomes significant. We suggest that a sparsity level around 50% strikes a favorable balance between model efficiency and performance in practical scenarios.

Q2: Comparison with one-shot methods.

We acknowledge that, unlike one‑shot methods such as Wanda or SparseGPT, our bilevel framework incurs additional runtime. However, our entire process still completes within a few hours on standard hardware, which is always affordable in downstearm applications.

Importantly, model compression is a one‑time offline cost, where peak GPU memory usage often outweighs total runtime in practical deployments. Our method uses no more memory than one‑shot approaches but delivers superior accuracy and inference speedups, making the extra time investment worthwhile. The results of compression time are summarized in Table R3. Even QR, a simplified version of our method, still outperforms SparseGPT and Wanda under comparable time cost.

Table R3. Compression time, Phi-3 mini 60% prune rate

Q3: Generalization across architectures.

Our main experiments focus on Phi and LLaMA2/3 models. Since our method performs optimization at the matrix level without relying on specific model architectures, it is generally applicable to other network structures as well. To further address the reviewer’s concern, we have conducted additional experiments on Qwen2.5-3B, with results shown in the table below.

Table R4. Additional experiments on Qwen2.5 3B

Q4: Fine-tuning requirements.

To ensure a fair comparison aligned with current research paradigms, we did not perform any task-specific fine-tuning for any of the compression methods and instead evaluated their zero-shot performance. We would like to emphasize two points:

• First, at a sparsity level of 50%, the compressed models are able to retain over 95% of their zero-shot performance, making fine-tuning unnecessary.
• Second, fair comparisons under fine-tuning settings are non-trivial. While our method can efficiently fine-tuning compressed model using low-rank component, methods like Wanda and SparseGPT require careful structural modifications, which fall beyond the scope of this work. We leave this direction for future exploration.

Q5: Hardware efficiency.

We appreciate the reviewer’s comment and agree that presenting inference speedup over different platforms is important. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results. To address this, we followed the ATP[r1] protocol and conducted GPU inference tests using nm-vllm. The results are presented in Table R5. Compared with unstructured methods, our approach, Low rank & Sparse, leads to more practically meaningful GPU inference acceleration.

Table R5. GPU inference speedup LLaMA2-13B, on a single 80 GB A100 GPU

References

[r1] Lujun Li et al. "Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models."

Thank you very much for your response and encouraging feedback on our work. Do you have any further questions or comments? We would be more than happy to engage in further discussion at your convenience.

W1: Theoretical guarantees, optimality conditions and so on.

Thank you for pointing out this limitation. We provide a theoretical convergence analysis with optimality conditions in our response to your Question 2. We will incorporate these discussions into the revised version if accepted.

W2, W4 & Q1: Hyperparameters' sensitivity.

We acknowledge that our bilevel framework introduces several hyperparameters. However, we would like to clarify following facts to show that our algorithm is not sensitive to these hyperparameters.

• In fact, we tuned these hyperparameters on Phi-3 mini and applied the same configuration across all other model experiments. Besides, parameters such as the initialization of the truncated Gaussian mean $\mu$ and the number of inner-loop iterations are consistent with those used in OATS.
• We also conduct ablation studies on these key hps, and the results are presented in Table R1 below.
• Moreover, theoretical analysis (see our response to your Question 2) shows the dependency of convergence on the parameters such as $\gamma$.

Table R1. Ablation study of hyperparameters Phi3-mini with 50% prune rate, wikitext2 PPL

W3 & Q3: Large-scale applications.

Thank you for pointing out this concern. Due to the space limit, please refer to our response to reviewer ZdES's W2 & Q1.

Q2 Theoretical analysis:

We would like to deliver a formal form of our theorem below by adding some assumptions following [r1].

Theorem For bilevel problem with the form as:

We assume $\Phi(\theta)$ is $L$-smooth, and the policy gradient variance satisfies:

Let $\eta < 1/L$ and define the gradient mapping $\mathcal{G}^t$ at iteration $t$ as:

then when $T \rightarrow \infty$, we have:

Remark We would like to point out the following:

• We give this theorem just to show that our algorithm behaves similarly to general projected/proximal stochastic gradient descent algorithms for one-level optimization problems (e.g., [r2]), rather than to characterize how fast our algorithm converges. Therefore, we do not consider variance reduction techniques, which are out of scope for this work.

• $\tilde{\sigma}^2$ is the variance of our gradient estimator PGE with a single sampling of (s,$\kappa$). It can be reduced using multiple sampling, larger batch sizes and other variance reduction techniques, hence the LHS of Equation (1) can converge to a small value.

• As shown in line 912 of Appendix D.3, $\tilde{\sigma}^2$ becomes small when $\gamma$ is small. In our experiments, $\gamma$ is gradually reduced to a smaller value to facilitate convergence.

We need the following lemmas about the properties of the projection operator, which can be found in [r2].

Lemma 1 (Firmly Nonexpansive Operators)
Let $\mathcal{C} \subset \mathbb{R}^d$ be compact and convex. For any $\mathbf{u}, \mathbf{v} \in \mathbb{R}^d$:

Lemma 2
Let $\mathcal{C} \subset \mathbb{R}^d$ be compact and convex. Then for any $\mathbf{c} \in \mathcal{C}$ and $\mathbf{u}, \mathbf{v} \in \mathbb{R}^d$:

Proof of Theorem: We define:

Our method updates $\theta$ as:

Let the stochastic and deterministic gradient mappings be

we can have

Therefore, we can get

Thus, we obtain

Now, we turn to analyze the term $\langle \delta^t, \mathcal{G}^t \rangle$ as follows:

For $\lVert \delta^t \rVert^2$, we have

Combining inequalities (2), (1), and (3), we have

Finally, we bound $\mathbb{E} \lVert \mathcal{G}^t \rVert^2$ as follows:

Combining inequalities (5) and (4), as $T \rightarrow \infty$, we obtain

$\blacksquare$

References

[r1] Pedregosa et al., "Hyperparameter optimization with approximate gradient."

[r2] Ghadimi and Lan, "Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization."

W1: Novel framework and well-established components.

We thank the reviewer for acknowledging the novelty of our framework.

While the individual components are well-established, our contribution lies in designing a collaborative mechanism and solving strategy that coordinate them under a bilevel framework, which, to the best of our knowledge, is the first of its kind in model pruning. We believe such collaboration framework would be effective and helpful to developing high-precision pruning methods for LLMs, especially in the scenario with limited computational and data resources.

Moreover, beyond the general framework, we incorporate several specific design choices to enhance training efficiency. For instance, we adopt a truncated Gaussian prior, which effectively mitigates the risk of gradient explosion (see Section 4.2). This issue can arise when using a standard Gaussian distribution, as its gradient contains a $1/\sigma^2$ term. To ensure convergence, one typically needs to reduce $\sigma$ to concentrate the sampling distribution, but this can lead to unstable gradients. The truncated version addresses this problem by allowing us to concentrate samples via truncation while keeping $\sigma$ fixed, thereby maintaining gradient stability.

W2: Scalability concerns.

Thank you for pointing out this concern. Due to the space limit, please refer to our response to reviewer ZdES's W2 & Q1.

W3: Limited improvement on zero shot and speedup.

Regarding zero-shot accuracy, we would like to clarify two key points.

• Zero-shot task is relatively easy. At low sparsity levels (e.g., below 50%), the performance degradation caused by pruning is generally minor—even for baseline methods, the post-pruning performance remains close to that of the original model. This suggests that the room for further improvement is quite limited, making it nearly impossible to achieve significant performance gains under such settings.

• Our method demonstrates clear and significant improvements under high sparsity levels (60–70%), as shown in Table 2 of the main paper and Table R1 below.

Table R1. Performance comparison under high prune rates.

Regarding inference speedup, we argue that

• We present the results in Table 4 to illustrate the additional speedup gained from the low rank and sparse structure; specifically, low rank matrices are generally more hardware-friendly for acceleration, rather than highlighting their overall superiority in speedup.
• At the same compression rates, the compressed models obtained from different pruning strategies usually yield comparable inference speedups, as the additional speedup derived from the unique matrix structure cannot be excessively high.
• Consequently, our objective is to achieve higher accuracy than the baselines while maintaining the same compression ratio, as demonstrated by the results presented in the main text.

W4 & Q3: Hyperparameter sensitivity.

We acknowledge that our bilevel framework introduces several hyperparameters. However, we would like to clarify following facts to show that our algorithm is not sensitive to these hyperparameters.

• In fact, we tuned these hyperparameters on Phi-3 mini and applied the same configuration across all other model experiments. Besides, parameters such as the initialization of the truncated Gaussian mean $\mu$ and the number of inner-loop iterations are consistent with those used in OATS.
• We also conduct ablation studies on these key hps, and the results are presented in Table R2 below.
• Moreover, theoretical analysis (see our response to reviewer tY2u's Question 2) shows the dependency of convergence on the parameters such as $\gamma$.

Table R2. Ablation study of hyperparameters Phi3-mini with 50% prune rate, wikitext2 PPL

W5: Comparisons with GBLMpruner & ATP.

Thank you for pointing out these recent works in model compression. We present the comparison results in Table R3.

It is worth clarifying that ATP originally uses a customed and simplified version of the lm_eval package (based on version 0.3.0). Since we use an updated version (0.4.2) consistent with OATS in our main experiments, there may be slight discrepancies in evaluation results, which is also mentioned in OATS's Appendix A.12. To ensure fairness, we re-evaluated the perplexity on Wikitext2 of our methods using the version adopted in ATP's implementation.

Due to our learned sparsity allocation and the integration of low-rank matrices, our method outperforms both GBLM-Pruner and ATP across all settings.

Table R3. Comparison with GBLM and ATP on LLaMA2-7B. "-" means missing results in the original paper.

W6 & Q1: GPU Inference speedups.

We appreciate the reviewer’s comment and agree that presenting GPU inference speedup is important. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results. To address this, we followed the ATP protocol and conducted GPU inference tests using nm-vllm. The results are presented in Table R4. Compared with unstructured methods, our approach, Low rank & Sparse, leads to more practically meaningful GPU inference acceleration.

Table R4 GPU inference speedup LLaMA2-13B, on a single 80 GB A100 GPU

W7 & Q2: Convergence Analysis.

Due to space limitations in the responses, we provide a detailed convergence analysis of the bilevel framework in our response to Reviewer tY2u's Question 2. We will include this analysis in the appendix of the revised version if accepted.

Theoretical analysis demonstrate that a smaller value of $\gamma$ leads to reduced gradient variance, which in turn facilitates faster convergence of the optimization process.

Q4: Time taken compared to one-shot methods:

We acknowledge that, unlike one‑shot methods such as Wanda or SparseGPT, our bilevel framework incurs additional runtime. However, our entire process still completes within a few hours on standard hardware, which is always affordable in downstearm applications.

Importantly, model compression is a one‑time offline cost, where peak GPU memory usage often outweighs total runtime in practical deployments. Our method uses no more memory than one‑shot approaches but delivers superior accuracy and inference speedups, making the extra time investment worthwhile. The results of compression time are summarized in Table R5. Even QR, a simplified version of our method, still outperforms SparseGPT and Wanda under comparable time cost.

Table R5. Compression time, Phi-3 mini 60% prune rate