DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs

We thank the reviewer for the insightful questions and suggestions on error bound, quantization compatibility, and different LLM architectures. We have tried our best to address all of them in our rebuttal.

Q1: Since dual sparsity has a closed-form and the accuracy is usually acceptable, I wonder if there is any more concise bound of error in terms of numeric values or higher-level semantic information.

Thank you for this important question and the reference. We provide a concise improvement bound quantifying DuoGPT's guaranteed loss improvement over SparseGPT below.

Theorem (DuoGPT Improvement Bound): Under activation sparsity level $p_x$, DuoGPT achieves guaranteed loss improvement over SparseGPT:

${\Delta}L = L_{\text{SparseGPT}} - L_{\text{DuoGPT}} = r \hat{X}^T H^{-1}_{-p} \hat{X} r^T$.

Then, the loss improvement satisfies the lower bound:

$\geq \dfrac{\alpha p_x \sigma_r^2 C_W^2 m}{\lambda_{\max}(H)}$.

where $\alpha > 0$ is the stability constant measuring spectral gap preservation, $\lambda_{\max}(H)$ is the maximum eigenvalue of the Hessian $H = \hat{X} \hat{X}^T$, $\sigma_r > 0$ is the activation residual magnitude constant, $C_W$ is the weight norm bound that satisfies $\||W\||_F^2 \leq C_W$, and $m$ is the calibration batch size.

Proof Sketch:

The key insight is that the weight update rules of SparseGPT and DuoGPT differ only in the residual correction term (Equation 3). Therefore we have the loss of SparseGPT as

$L_{\text{SparseGPT}} = || -\frac{w_p}{H^{-1}_{pp}}||^2_F$, and the loss of DuoGPT from the Equation (20) in the appendix.

When expanding both loss functions and subtracting $L_{\text{DuoGPT}}$ from $L_{\text{SparseGPT}}$, all terms cancel except one quadratic form: $r \hat{X}^T H^{-1}_{-p} \hat{X} r^T$.

Since $H^{-1}_{-p}$ is positive definite, this guarantees $\Delta L \geq 0$ always.

For the lower bound, we use the fact that $H^{-1}_{-p}$ is obtained from $H^{-1}$ via Gaussian elimination, which follows the spectral bound that

$\lambda_{\min}(H^{-1}_{-p})$ is greater or equal to

$\frac{\alpha}{\lambda_{\max}(H)}$. After applying the quadratic form bound where

$r \hat{X}^T H^{-1}_{-p} \hat{X} r^T$ is greater or equal to

$\lambda_{\min}(H^{-1}_{-p}) \cdot \||r \hat{X}^T\||^2$. Using activation sparsity properties yields our result.

Empirical Validation: Our bound predicts meaningful improvement that scales linearly with activation sparsity $p_x$. Experimentally, we observe consistent improvement over SparseGPT across all models and sparsity levels in Table 1 and Table 3a, confirming the bound's validity. The linear scaling with $p_x$ explains why higher dual-sparsity yields greater improvements.

We will add discussion of the bound to the revised version of the paper. The complete formal proof will be included in Appendix of the revised manuscript. We will make sure to reference the provided paper as our theoretical backup.

Q2: Compatible with weight quantization?

Thank you for this question! You are correct about the orthogonal relationship between quantization and OBC-based pruning - they operate on different aspects of model compression and can be combined effectively.

We evaluated the combination of INT8/INT4 uniform weight (group size 128) quantization with 50% dual-sparse DuoGPT on LLaMA-2-7B and 13B (WikiText2 PPL).

DuoGPT maintains its performance advantage over SparseGPT across all precision levels, demonstrating successful orthogonal combination. Notably, our method shows better robustness to INT4 quantization, suggesting that the activation-sparsity-aware calibration helps preserve model quality under aggressive compression.

This multi-dimensional compression approach is very promising for practical edge deployment-combining INT4 precision with 50% dual-sparsity achieves roughly 6% model density for SRAM loading, a compression ratio difficult to attain through any single compression method alone.

Q3: Different LLM architectures

We acknowledge this important concern about different LLM architectures. To address this, we have evaluated DuoGPT across multiple model families including Mistral, Qwen2, and OPT (as shown in our response to Reviewer B, Q1). The results demonstrate consistent improvements across all tested architectures, confirming that our theoretical framework and empirical benefits generalize beyond LLaMA models.

We thank the reviewer for the valuable comments and suggestions on additional experiments. We have tried our best to address all of them in our rebuttal.

Q1: The current experimental evaluation is limited to the LLaMA architecture.

Thank you for this important suggestion. We agree that demonstrating generalizability across architectures is crucial. We have evaluated DuoGPT on multiple model families beyond LLaMA, showing consistent improvements across different architectures. We tested on Mistral, Qwen2, and OPT with varying model size in the table below with 50% dual-sparsity on WikiText2 PPL.

Q2: Latency results.

Thank you for this important suggestion as wel. We provide latency analysis below, noting that our method specifically targets single-batch decoding scenarios where activation sparsity can be effectively exploited through GEMV operations—the critical use case for interactive applications.

In the table below, we measured end-to-end latency (% of reduction compared to dense baselines) across different context lengths using the setup described in Appendix B.2, showing consistent improvements that increase with model size (24.6% average for 7B vs 29.2% for 13B). This scaling behavior reflects the memory-bound nature of single-batch decoding, where our dual-sparsity approach effectively reduces SRAM loading costs.

We note that significant system optimization opportunities exist for dual-sparse LLM workloads, and we hope this work opens promising avenues for the ML-systems community to explore.

Q3: GPU kernel implementation.

Thank you for bringing up this important implementation concern. We want to clarify our current kernel support and future work directions:

We have implemented end-to-end triton-based GPU kernels for exploiting activation sparsity, which enables the latency measurements reported in our above response and the throughput results in the main paper. The key idea of the kernel is to use triton.language.load to selectively load weight rows based on a bitmask representing the activation sparsity pattern. This demonstrates a lower bound of the real-world efficiency gains from our dual-sparsity approach.

The remaining kernel to implement is for compressed storage/loading of unstructured sparse weights. However, this weight compression kernel is orthogonal to our current hardware performance results, as the primary SRAM loading efficiency gains come from the activation sparsity exploitation during GEMV operations.

Full kernel support for weight compression represents promising future work that would provide additional HBM loading reduction benefits on top of our current demonstrated improvements. We view this as an exciting direction that could further amplify the efficiency gains of dual-sparse LLM workloads.

We thank the reviewer for the valuable comments and insightful suggestions and have tried our best to address all of them in our rebuttal.

Q1: Comparison with CATS

Thank you for the suggestion. We acknowledge that CATS is a pioneering work on activation sparsity in LLMs and provide a detailed comparison below.

Methodological Differences:

CATS focuses on output-activation sparsity in gated MLPs by applying thresholds after non-linear activation functions (e.g., zeroing $\text{SiLU}(xW_{\text{gate}})$ neurons below a threshold). This approach targets SRAM savings specifically for $W_{\text{up}}$ and $W_{\text{down}}$.

In contrast, DuoGPT focuses on input-activation sparsity by thresholding the inputs to each layer. This design allows activation sparsity to be applied across all linear layers in LLMs, not just FFN blocks.

Performance Comparison:

We evaluate both methods on five downstream tasks using LLaMA-2-7B at 40% global sparsity. DuoGPT consistently outperforms CATS across all tasks, as shown below.

Q2: Why not ReLU?

Thank you for the question. While ReLU can indeed induce activation sparsity, our approach offers several key advantages:

1. ReLU provides output-activation sparsity only for FFN blocks, while our input-activation sparsity works across all linear layers in LLMs.

2. Our method requires no fine-tuning, operating in a one-shot calibration fashion. ReLU replacement requires extensive fine-tuning (e.g., 30B tokens for LLaMA-7B [1]).

We further compare DuoGPT with the ReLU-based method [1] at 22% global sparsity on LLaMA-2-7B below.

Q3: Comparison with structured + unstructured pruning

Thank you for pointing out the STUN reference—we will include it in the related work section. However, we would like to clarify a key misunderstanding: DuoGPT does not combine structured and unstructured pruning.

Key Methodological Distinction:

- STUN combines actual structured pruning (e.g., 5% for non-MoE models) with unstructured pruning (often >50%). While this achieves good accuracy, the unstructured component remains difficult to accelerate efficiently on SIMT architectures.

- DuoGPT, by contrast, performs only activation-aware unstructured weight pruning during calibration. At inference time, we reinterpret activation sparsity as a form of dynamic structured sparsity during GEMV operations. This allows for GPU acceleration without the accuracy degradation typically seen in actual structured weight pruning.

Core Innovation:

We leverage the insight that, in single-batch decoding (GEMV), activation sparsity naturally behaves like structured sparsity: only the weight rows corresponding to non-zero activations are accessed. Building on this, our main technical contributions include extending the OBC framework with a novel closed-form solution that rigorously accounts for activation sparsity effects using asymmetric residual correction (Eq. 3), and develop an efficient GPU implementation that scales to 70B+ models in under 140 minutes.

Performance Comparison:

On 5-shot GSM8K with LLaMA-2-7B, DuoGPT achieves comparable accuracy to STUN (e.g., 7.2% with 47.5% weight sparsity and 15% activation sparsity) while offering 3× greater acceleratable portion (15% activation sparsity vs. 5% structured sparsity in STUN), demonstrating the practical effectiveness of our dynamic structured sparsity approach.

Q4: Dependent on additional computation

We appreciate this concern and would like to clarify two important points:

1. No Causality Between Compute and Effectiveness:

One good example would be that the additional calibration time is a constant overhead, independent of the sparsity level. Meanwhile, the effectiveness of DuoGPT increases with dual-sparsity level. As shown in Table 6, our perplexity (PPL) improvement on LLaMA-2-7B grows from 0.5% (at 30% dual-sparsity) to 12.5% (at 65% dual-sparsity)—all at the same calibration cost.

2. Expected Behavior for Large Models:

Larger models are known to be more resilient to compression. For example, Wanda [2] reports PPL improvement dropping from 0.09 (7B) to 0.00 (70B) compared to SparseGPT. Our observed improvement of 0.07 on LLaMA-2-70B is thus reasonable and consistent with prior work.

Importantly, the calibration cost is a one-time offline expense that scales sublinearly with model size. It is negligible compared to training costs and enables efficient inference over the model’s entire deployment lifecycle.

Q5: Isolate the effect of residual correction without activation sparsity

Thank you for the suggestion. We conducted the requested ablation to isolate the effect of the residual correction effect by evaluating two scenarios: (1) weight-only pruning (no activation sparsity), and (2) dual-sparsity (with activation sparsity).

The results demonstrate the following:

1. Residual correction is effective on its own: DuoGPT outperforms SparseGPT even in the weight-only setting, showing that the second term in Δw (Eq. 3) provides a clear benefit.

2. Activation sparsity amplifies the gain: When activation sparsity is included, the magnitude of improvement increases by 2.9×, 1.8×, and 2.6× respectively across the three models.

These results confirm that our closed-form solution adapts effectively: the residual correction term compensates for calibration error from both weight pruning and activation sparsity.

The table below shows the results of the ablation study isolating residual correction effects (50% sparsity, WikiText2 PPL)

References

[1] Mirzadeh, Iman, et al. "ReLU strikes back: Exploiting activation sparsity in large language models." arXiv preprint arXiv:2310.04564 (2023)

[2] Sun, Mingjie, et al. "A simple and effective pruning approach for large language models." arXiv preprint arXiv:2306.11695 (2023).